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ADSTRACT 

This program is designed to train individuals to 
develop and evaluate educational products by weans ot a consortium 
composed of agencies and individuals in Southern California with 
expertise in cne or more aspect of product development and product 
evaluation. The trainees would include degree and non-degree 
candidates, and the program would provide training foi individuals 
wishing to acquire specific skills and for agencies wishing to 
promote training for a staff sub-group. By focusing upon the training 
of outstandingly competent product developers and product evaluators, 
the agencies collaborating to design the training program believed 
that they could best contribute to the improvement of the country's 
schools. The pri cipal site would be a training laboratory located 
near the UCLA campus, and the basic year-round training weald be 
supplemented by specialized institutes and short courses. The 13 
agencies in the consortium are identified and described. Tue various 
programs (one year, short term, graduate degree- rel ated , and master 
of c-rts) are outlined, together with the instructional objectives an i 
measurement procedures. Additional material includes vitae of the 
program staff and copies of correspondence concerning the development 
of the program. (?19rt) 
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BRIEFING SUMMARY 



New Design for Training 



Prime contractor for operational phase : University of California, 
Los Angeles 

Project D irector for operational phase : Dr. W. James Fophan, 
Professor, Graduate School of Education, University of California, 

Los Angeles 

Principal participants in op_eratlonal pha se: 



Institutions 

American Tape Duplicators 

BFA Educational Media 

Center for the Study of Evaluation 

Consolidated Film Industries 

CTB/McGra*-Hill 

Educational Development Corporation 

Graduate School of Educatitn, UCLA 

Institute for Educational Development 
KCET Television Station 
Los Angeles County Superintendent 
of Schools Office 
Theater Arts Department, TJCLA 



In dividua ls^ 

Donald E, Anderson, Richard L. Zveig 

Grant R, Cary, Rex Malcolm 

Marvin C. Alkin 

Sidney p. So low 

Joseph Dionne 

Patricia Harrison, George Rosato, 
Adrian B. Sanford 
Eva L. Baker, Evan R. Keislar, 

John D. McNeil, W. Janes Pophaa 
Robert T. Filep / 

Maynard Orrae 

Robert Gerletii 
Walter R. Kingson 



Major manpow er needs being addressed . This program is designed to 
prepare educational developers and evaluators of already developed 
educational products. The focus of the program is primarily on 
development training, with its secondary emphasis being upon the 
training of evaluators who can appraise the quality of already prepared 
instructional materials 

Unique features of rationale , content , and process of the proposed 
design : The training program design features a results-orientation for 
both kinds of specialists being prepared, namely, product developers 
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and product evaluators. Professionals vri.ll ue prepared v/ho are 
particularly cognizant of the e ffects of instructional materials 
with which they are working. They will be less enamored of 
instructional processes for their own sake than with the results 
those processes yield, oituated in the Southern California area 
where a number of professional educational materials developers 
are located, particularly in the film and television industries, 
the training program will be operated by a collaborative consortium 
of eleven egencies. Each of these agencies is capable of making a 
unique contribution to the training enterprise, 

A special Training Laborator y will be located off campus near 
UCLA, thereby capitalizing on the resources of that institution and 
permitting collaborative degree-granting programs. The La boratory 
will be sufficiently far removed, both physically and instructions Ily, 
to permit the program to be truly innovative but, above all, focused 
on getting results. Extensive praticum assignments will also be 
arranged on site within the various agencies conducting the training 
program. 

The quality of various phases of the training program will be 
assessed by customary evaluation techniques and by uniquely devised 
performance tests which will provide an opportunity for the product 
developers and the product evaluators to display their competencies 
in simulated situations which approximate the tasks they will be 
called upon to perform at the conclusion of their training. 
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DESCRIPTION OF ACTIVITIES CONDUCTED PURSUANT TO 
RESEARCH CONTRACT CEG-0-70-4765(520) 



A Training Program For Developers and Evaluators 
Of Educational Products 



This report will describe the operations associated with conduce 
of a contract research project for the U.S. Office of Education. The 
chief product of the contract was an extensive design for a training 
program to prepare educational developers and product evaluators. That 
design, submitted separately to the Office of Education, is presented 
in a document entitled A Training Program for Developers and Evaluators 
of Educational Products* . The report contained in the following pages, 
however, will deal only with the procedural, not the substantive, 
aspects of that project. 

A Request for Proposals Arrives 

Upon receipt of RFP Number 70-12 which was circulated by the De- 
partment of Health, Education, and Welfare in Spring, 1970 to solicit 
proposals to design new patterns for training particular kinds of edu- 
cational researchers, five professors affiliated with the Graduate 
School of Education, University of California, Los Angeles discussed 
the wisdom of responding to the RFP, both in terms of their own interests 
as well as the potential local resources which might be drawn upon to 
carry out a major training program, should one be funded. 6 positive 
decision was reached and a lengthy series of discussions U .. place re- 
garding what kind of proposal to submit, what kinds of training strat- 
egies to emphasize and, most curcially, what kinds of educational re- 
searchers to train. The reasons for the positive decision, as well as 
a choice to emphasize the training of educational developers and educa - 
tional produc t evaluators, is perhaps best summarized in the introduc- 
tion to the proposal which was prepared in response to RFP 70-12 and may 
be useful to the reader, for it accurately depicts why these five indi- 
viduals were drawn to the particular kind of training program which they 
recommended. Accordingly, that section of the proposal is presented 
below: 



Almost five years ago a group of individuals associated with 
the University of California, Los Angeles became convinced that to 
produce really dramatic improvements in the quality of American 
education, the effectiveness of the materials used for instruction 
would have to be drastically increased. Such a commitment emerged 
quite naturally from thiE group’s experience during the early 
sixties with the programmed instruction approaches. A formidable 
collection of researchers concerned with programmed instruction 
variables had assembled at UCLA, including Arthur A. Lumsdaine, 
Susan Meyer Markle, Evan R. Keislar, John D. McNeil, and W. James 
Popham. Their research efforts yielded useful insights regarding 
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* Graduate School of Education, University of California, Los Angeles, 
December, 1970. 
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what in the mid-196Q's was considered programmed instruction, i.e., 
small step programs ir the Skinnerian tradition. Yet it became 
clear that the trial-revisj on strategy which was used with pro jammed 
instruction, not to mention a host of related tactics, e.g., use of 
measurable objectives, was equally applicable to any reproducible 
set of instructional materials. Thus when Lumsdaine 1 and Markle*- 
urged that a broadened conception of a "program" be employed so that 
it included any set of replicable instructional events, their UCLA 
colleagues readily concurred. 

But the programmed instruction movement, as a vehicle for pro- 
moting widespread improvements in American education, began to lose 
its luster. It had become evident that commercial publishers could 
not, or would not, expend the considerable funds necessary to nurse 
large scale programmed instruction sequences through the costly 
trial-revision cycles they needed for demonstrable effectiveness. 

Few of the early programmed instruction entrepreneurs were still in 
business. The table stakes to play the materials development gams 
were too high for most. An infant technology vas growing, but 
massive financial resources were required. Yet when the prospects 
seemed most bleak, the situacion brightened unexpectedly. The 
federal government was going to support development activities 
through the U.S. Office of Educations support of such agencies as 
the newly established research and development centers and regional 
laboratories. 

Yet, as many of these new institutions emerged they did not, 
to the chagrin of our group at UCLA, emphasize the development of 
educational products. Instead, there vas a plethora of classic 
educational research projects or, perhaps, efforts to promote certain 
innovations, e.g., microteaching, interaction analysis. Few centers 
or laboratories focused their major programs on development. For we 
had been Hire alistically optimistic* How could there be many full 
product development enterprises? There weren't enough competent 
development specialists to go around. This deficiency had to be 
rectified. 

We had to promote a new form of specialization in which large 
numbers of educators wouJd acquire the competencies needed for the 
development of validated instructional products. Popham' s 1966 
article^ advocating such a specialty stimulated several UCLA col- 
leagues, chiefly Eva L. Baker, to develop a new doctoral level 



HiUmsdaine, A. A. ’"Educational Technology, Programmed Learning, and 
Instructional Science." Theories of Learning and I nstr uction. Sixty- 
third Yearbook, National Society for the Study of Education, Part I. 
Chicago: Distributed by the University of Chicago Press, 1964, p. 385. 
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Markle, Susan M. "Empirical Testing of Programs." Programed In- 
struction. Sixty-sixth Yearbook, National Society for the Study of 
Education, Part II. Chicago: Distributed by the University of Chicago 

Press, 1967, p. 104. 

Popham, W. James. "Product Research: A New Curriculum Specialty. " 

Educational Leadership, March, 1966. p. 507-513* 
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training program which received one of the initial Title IV ESEA 
research training program grants. For several years the UCLA 
Product Research Training Program was the only graduate level 
training program of its kind in the United States, ^ that is, it 
was the only graduate training program with the primary mission 
of training educational developers. 

Several^ of the UCLA group, in addition to their responsibili- 
ties in the Product Research Training Program, developed full or 
part-time working relationships with the Southwest Regional Labora- 
tory for Educational Research and Development (SWRL), one of the 
few regional laboratories w‘.th a heavy cccmitment to the develop- 
ment of educational products* During the early phases of SWRL’s 
growth there was considerable activity involving the identification 
of critical skills required for the development of educational 
products. In particular, Drs. Baker and Popham participated ac- 
tively in these enterprises, leading in 19&7 to the preparation of 
a number of self-instructional products designed for training pro- 
spective educational developers. For example, Rules for the Devel - 
opment of Instructional Product s^ represented an effort to systema- 
tize and transmit the primary skills needed at key points in the 
development enterprise. But as SWRL's project c 012 ml t me nts expanded 
on several fronts, the emphasis on staff training was diminished. 
Little in the way of analyzing the skills required in development 
or in preparing materials to promote them has occurred at SWRL for 
the past few years. There is a :eed, accurately reflected in the 
R?P for which this proposal is a response, to devise some new ap- 
proaches to the identification and cndificat on of development skills 
and to design training patterns to prepare large numbers of profes- 
sionals who possess such skills. 

During the past several years the UCLA Center for the Study of 
Evaluation (CSE), particularly through the efforts of its current 
Director, Marvin C. ATkin, has been increasingly interested in ques- 
tions associated with the evaluation of educational products and of 
programs in which such products are used. It was perhaps predictable 
that the proximity of two groups concerned with (l) the development 
of educational products and (2) the evaluation of educational pro- 
grams would find numerous arenas of mutual interest. Such has been 
the case, particularly as a function of Frofessor A Ikin' s CSE leader- 
ship. 



According to a nationwide survey conducted by officials of the McGrav 
Hill Inc. Publishing Company. 

^Professors Baker, Keislar, McNeil and Popham. 

^Popham, W. James and Baker, E. L. Rules for the Development of Instruc - 
tional Produces . Southwest Regional Laboratory for Educational Research 
and Development: see also Baker, E. L. "Design Specifications: Objec- 
tives and Prototype Items." Developing Instructional Products : A Col - 

lection of Working Papers mid Training Docurants, Southwest Regional 
Laboratory, October, 1968* 
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We have, for example, recently established a doctoral program 
focused on educational evaluation in the UCIA Graduate School of 
Education. This program, one of only a few such across the nation, 
is staffed by a number of professors who jointly function as mem- 
bers of program groups involved in the product research doctoral 
program as well as the educational evaluation doctoral program. 

Once more because of Professor Alkin's influence, CSE has 
during the last year beccoe far more heavily involved in the devel- 
opment of educational products to prorate the acquisition of shills 
needed by evaluators. Indeed, a current project of CSE capitalizes 
on the efforts of seven of the product research trainees as they 
develop and field test instructional products to be used in one of 
the Center's research and dissemination enterprises. The Center is 
net only preparing a variety of replicable instructional products 
to be used by evaluators, 7 it is particularly interested in ques- 
tions of how to evaluate such products both summatively and foma- 
tively, that is, both after they are completed and during their 
development . 

In retrospect, then, it is apparent how individuals such as 
Dre. AUcin, Baiter, Keislar, McNeil and Poph&m would be vitally in- 
terested in designing new ways to train developers and evaluators 
of educational products. They are convinced of the potentially 
beneficicl impact of such produces. They have been actively in- 
volved in inquiry and training efforts associated with development 
end evaluation. In concert with colleagues from other agencies 
possessing personnel and situational resources of value in the con- 
templated training activities, they are anxious to get under way. 

The Proposal is Approved, Work Ccmraences 

In mid-June a contract was awarded to support the design activities 
of our group. Imaediately the five previously mentioned individuals met 
to explore strategies for consummating the project since it had to lead 
to a final report that was, in essence, a proposal to support a major 
training program for product evaluators and educational developers. 

These early s umme r discussions were marked primarily by questions regard- 
ing which was the best method to follow in putting together a training 
design. Since the RFP had stipulated that the training program was to 
consortium-based, should potential members of the training consortium be 
invited in at the outset to participate in the discussions? Should the 
five UCIA professors devise training scheme alternatives and then take 
these to possible consortium members? If consortium agencies were to be 
involved integrally in the development of the proposal, we believed that 
a decision regarding their participation should be made immediately. 



^Alkln, Marvin C. ejt al. Simulate d Evaluation Exercises Instructor's 
Manual, Center for the Study of Evaluation, UCIA, Report No. 49 ," May/ 
1969 ^ 
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After exhaustive consideration, it was decided that at least for the 
early period of analysis it would be preferable for us to work alone, 
involving other individuals as informal consultants and advisors, rather 
than establish a formal consortium organization early in the game. As 
it turned out, this decision had some advantages and disadvantages. 

On the positive side, this insular strategy enabled our group to 
engage in an unconstrained analysis of what was needed nationally to 
improve the quality of training for educational developers and product 
evaluators. We did not have to worry about the local availability of 
resources, we could look at the entire country as our training arena. 
Abandoning parochial constraints seemed to permit more imaginative pro- 
posals during our discussions. We devised a scheme which was truly 
nationwide in orientation, consisting in essence of a consortium that 
was a confederation of all those individuals and agencies who possessed 
resources to supply training in our two chosen specialties. We envisaged 
our scheme as an attempt to upgrade the entire quality of developer and 
evaluator training throughout the United States. While agencies within 
the Southern California region would be Involved in such a consortium, 
as would any other agency in the United States having sufficient training 
resources, there was no particular need to involve local groups inten- 
sively to work out the crucial elements of the consortium procedure. 

Thus, we would be able to save potential consortium members a consider- 
able amount of early planning time had our plan been approved. 

On the deficit side, however, it turned out that when we took our 
notion of a nationwide consortium to officials of the Office of Education 
for a required early September oral progress report, our scheme was not 
encouraged. We were advised that while there were meritorious elements 
in the plan, it was lokely that such a consortium notion could not be 
funded in the inmediate future • We were advised that it would be better 
for us to devise a locally based training consortium. Thus, we had es- 
sentially lost somewhat over two months devising a scheme which, while 
possibly suitable for future years, vas not appropriate for the project 
at hand, (incidentally, the most recent version of the working paper 
describing our conception of a nationwide consortium is provided in Ap- 
pendix A of the ^oapanion document, A Trai ning Program for Developers 
and Evaluators of Educational Products. 

Recruiting Consortium Members 

In view of these events, we had to intensify Immediately our work- 
ing relationships with potential consortium agencies, sccae of when had 
been contacted informally during the early summer, but none with a speci- 
fic request to participate in a locally operated training consortium. 

We devised the following brief document to outline possible roles of a 
consortium member in our training program. 
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Brief Description! A Possible Design For A Consortium- 
Based Training Program In Educational Product 
Development And Product Evaluation 

Purpose of the Trainin g P rogram * To train individuals who will 
(l) develop educational products and (2) evaluate educational 
products. 

Members of the Training Consortium . Agencies end individuals 
in the Southern California are- possessing expertise in one or 
more aspects of product development or product evaluation. 

Trainees . 'The individuals co be trained by the consortium 
would be of two kinds, degree and non-degree candidates. Pro- 
grams leading to advanced degrees (Masters and Doctorate) could 
be arranged in association with the Graduate School of Educa- 
tion, UCLA. The training program would provide training either 
for people who, as individuals, wished to acquire specific 
skills or for agencies (e.g., a regional educational laboratory) 
which wished to promote training for a staff sub-group. 

Nature of Training Program . A Training Laboratory located near 
the UCLA campus would be the principal site for training activ- 
ities, although specialized training would be conducted in 
other locale ■» (e.g., if equipment requirements dictate the de- 
sirability of on site training). A basic year round training 
program with two prime purposes, i.e., product development and 
product evaluation would be offered for a limited number of 
trainees (approx irately 20-30). Specialized institutes and 
short courses ranging from a few days to itev^ral weeks would be 
offered for approximately 100-150 individuals during the year. 

Nature of Consortium Member 1 s Partici p ation . Four different 
forma~of participation are envisaged. Consortium members would 
be encouraged to engage in as many of these as their interests 
and resources permitted: 

1. Membership on general ad visory committee which would 
meet every two months or so to offer overall advice 
regarding directions of the training entei prises. 

2. Visi ting Faculty Assignment at the Training Labora- 
tory wherein persona possessing particular competencies 
could, on a regular or e periodic basis, participate in 
the instructional effort of the Laboratory. 

3 . Supervising Internship Activities at the conbortiua 
member's own agency whereby one or more trainees would 
participate in an extended practicum assignment re- 
lated to product develojnent or product evaluation. 
(Nationwide practicum arrangements would also be 
sought, but not necessarily to be sponsored by consor- 
tium members. ) 



4. Conduct On - Site Training Components by offering rela- 
t ive ly "a ut on omou s t r a ini ng activities at the consortium 
member's agency. These training activities would be 
systematically related to the Training Laboratory's 
work, but could be conducted by consortium members* 
personnel. 



We then discussed this document through a series of individual 
meetings with representatives of the agencies listed below. Generally, 
these were face to face meetings, although in two cases extensive tele- 
phone conversations were used as a substitute. 



Agency 

BFA Educational Media 

CTB/McGraw-ilill 

Center for the Study of 
Evaluation 



Addrss 

11559 Santa Monica Boulevard 
Los Angeles, California 

Del Monte Research Park 
Monterey, California 

405 Hilgaru Avenue 
Los Angeles, California 



Consolidated Film 959 Seward Street 

Industries Hollywood, California 



Educational Development 
Corporation 

Graduate School of Educa- 
cation, University ot 
California 



220 University Avenue 
Palo Alto, California 

405 Hilgerd Avenue 
Los Angeles, California 



Institute foi Educational 
Development 

KCET 



The Los Angeles County 
Superintendent of Schools 

System Development 
Corporation 

Department of Theater Arts, 
University of California 



999 North Sepulveda Boulevard 
El Segundo, California 

1313 North Vine Street 
Hollywood, California 

115 West Washington Boulevard 
Los Angeles, California 

2500 Colorado Avenue 
Santa Monica, California 

405 Hilgard Avenue 
Los Angelej, California 
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All of the agencies contacted agreed to participate in the train- 
ing consortium. All were impressed with the importance of the mission 
and expressed considerable enthusiasm about the possibility of devising 
a training program which would prepare educational developers and prod- 
uct evaluators. Setae agencies offered to assist in the design activity 
more than others, as might have beer, expected. Frankly, the core 
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pl&ijiing group at UCLA vas reluctant to impose upon the terribly busy 
schedules of the people we had invited since these representatives are 
the very top individuals in their field. We had a choice of either 
going for mediocre people who might spend a great deal of time with us 
in designing the training program, o\ opting to secure the very best 
people who would be able to bo with us less frequently during the design 
phase. We chos f ; the latter alternative. It is important to note, how- 
ever, that these individuals all agreed to devote great attention to the 
training project should it be approved. It is a tricky situation one 
faces when trying to recruit a number of highly competent people to par- 
ticipate in designing a program which, during its creation period, is 
only probabilistically destined to exist. We were extremely gratified 
with the cooperation of individuals frcm the agencies identified above. 

The Total Group Convenes 

On October 14 the potential consortium members met as a group at 
the Sunset Canyon Recreation Center at UCIA. Although a number of the 
individuals attending this session already knew each other, there vas 
the necessity for introductions plus brief descriptions of the back- 
grounds and the primary mission of each agency. The probable role of 
each group in the training program was explored and information provided 
regarding the cunrent status of a draft statement regarding a proposed 
training design. Each group present was urged to describe in as much 
detail as possible the particular competencies which they could best 
provide for the training program. Each group represented was also asked 
to identify those staff members who would be participating in the project. 
A considerable amount of interchange occurred during this extended meet- 
ing, with various views being offered regarding how the training program 
should be devise i. Materials were identified which were to be subse- 
quently contributed by each agency for inclusion in the design report. 

After the October 14 meeting a series of individual conferences be- 
tween members of the project staff and the potential consortium members 
occurred, some by phone, most in person. In genera], these meetings 
were designed to describe in more detail the probable role of the con- 
sortium agency, or certain financial aspects of its arrangement with the 
consortium. 

When all the materials had been received from the consortium mem- 
bers a preliminary version of the final report of the project was sub- 
mitted to each agency on November 16 with the suggestion that modifica- 
tions be made and the document returned to the project director by 
November 23. Individuals in the consortium agencies were encouraged to 
contact the director by phone or set up a meeting for a more extensive 
discussion. 



The Final Draft 



While most agencies end individuals involved had few suggestions 
for changes, there was a fair amount of re-writing that occurred prior 
to transmitting the report to the VCIA Printing Office in late November. 
As indicated earlier, the resulting document of approximately 100 single 
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spaced pages describes the consortium-based training program devised as 
a result of this contract* 

Brief Needs Survey 

Earlier, in mid-September it was evident that information should 
be seemed regarding the possible locations vhere trainees vho had been 
prepared by the program could he placed, or vhere organizations existed 
vho would wish to train members of their staffs. We decided to contact 
representative groups of potential employers, as veil as those agencies 
possessing staffs vho might be in need of in-service education, in 
order to see whether there was indeed a strong interest in the program. 
Individual letters were sent to 50 Title III ESE1A coordinators in state 
departments of education, 50 chiefs of curriculum and instruction divi- 
sions in state departments of education, 50 producers of educational 
films, 50 publishers of textbooks and educational materials, 50 producers 
of programmed instruction materials, and 50 miscellaneous producers of 
educational materials, e.g., regional laboratories, large city school 
systems, etc. A copy of the letter transmitted to these individuals is 
included in the Appendix of this report, as are copies of 25 representa- 
tive responses. Responses to the 300 letters of inquiry through mid- 
November were arrayed as presented in Table 1. 



TABLE 1. Responses To Needs Survey Letter 



Type of Agency 


Number 

Sent 


Number 

Returned 


Interested 
in Program 


Not 

Interested 


ESEA Title III 
State Coordinators; 


50 


6 


6 


0 


State Education Department 
Curriculum and 
Instruction Chiefs: 


50 


29 


26 


3 


Producers of Educational 
Filina: 


50 


11 


9 


2 


Publishers of Textbooks 
and Educational Materials: 


50 


13 


10 


3 


Producers of Programmed 
Instruction Materials: 


50 


7 


7 


0 


Miscellaneous Producers of 
Educational Materials: 


50 


25 


21 


4 


Totals 


300 


91 


79 


12 




We believe the 87 percent positive responses to our inquiry, while 
only about one -third of those we transmitted, does provide an indication 
of the useful role which could be played by an effective training program 
for educational developers and product evaluators. A judgment regarding 
whether our group has been able to desiga a potentially effective program 
will have to be reached after examining that design: A T raining Program 

for Developers and Evaluators of Educational Products . 
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UNIVERSITY OF CALIFORNIA, LOS ANGELES 



BERKELEY • DAVIS • IRVINE • LOS ANGELES * RIVERSIDE * SAN DIEGO * SAN FRANCISCO 




SANTA BARBARA • SANTA CRUZ 



CENTER FOR THE STUDY OF EVALUATION 
UCLA GRADUATE SCHOOL OF EDUCATION* 
LOS ANCELES, CALIFORNIA 90024 



September 17, 1970 



A group of us here at UCLA, including Marvin Alkin, Eva Baker, Evan 
Keislar end John McNeil, have been awarded a contract by the U,S, Office 
of Education to design a training program for preparing (l) education 
product developers and (2) educational product evaluators. Near the end 
of the year ve will be submitting our program design to U.S.O.E. and, if 
approved, the training enterprise will be supported for at least a tnree 
year training period at a very substantial level. 

Since these individuals have a wealth of experience in training 
educational product developers and evaluators, we anticipate putting to- 
gether an outstanding program, UCLA has for the past several years of- 
fered the country's only doctoral level graduate program in instructional 
product development. In addition, the UCLA Center for the Study of Eval- 
uation is currently the major U,S, research and development agency devoted 
to inquiry regarding evaluation. 

There will probably be only three of these new training programs 
established, so they must definitely serve a national trainee clientele* 

I am writing you to determine whether your agency would wish to use the 
resources of the new training program eithei (l) to provide training in 
product development and/or product evaluation for members of your staff 
or (2) to employ newly trained individuals in either of these specialties. 

Let me be a little more specific about the training program we are 
planning. It will be operated by a consortia of agencies and individuals, 
most of whan will bo located in southern California, A staff drawn from 
this consortium will operate an off-campus training institute affiliated 
loosely with UCLA. The institute Will probabxy offer both a year-round 
basic training program as well as chorter, specialized courses (from a 
few days to several weeks). 
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Throughout the training activities there will be a dual focus on 
providing training (both beginning and advanced) for the following two 
specialties: 

Educational Devalonaent * The process oi preparing essentially 
replicable instructional materials or sequences which take 
responsibility for producing a given behavior change in speci- 
fied learners. Examples of educational products deve loped by 
this trial-revision sequence would include: printed self- 

instruction programs, highly systematized instructional pro- 
cedures, educational videotapes, filmstrips, etc# 

Educational Product Evaluation# The process of assessing the 
worth of already prepared educational products such as textbooks 
or films; that is, all of the materials which could be produced 
as a consequence of the work of the educational developer. 

Although of necessity I have been brief, is there any likelihood 
that if our training program is set up and functioning as of sunmer, 1971 
that your agency would wish to use the cervices of the program? If so, 

I hope you would be willing to write me indicating the nature of your 
training needs. 

Even though we are in the midst of working out ueteils of the new 
program, I will try to supply additional information if you wish. What 
I am attempting to do at the moment is simply to explore the current 
level of training requirements in our two fields of emphases# 

Sincerely, 



W# Janes Pophau 
Professor of Education 



WJP/re 
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DEPARTMENT OF EDUCATION 



22B W EOT STATE STREET 
R. O. »OX XO It 

TRENTON. NEW JERSEY OSttl 



DIVISION OF CURRICULUM AND INSTRUCTION 



October 7, 1970 
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Dr, W. James Popham, Professor of Education 

Center for the Study of Evaluation 

University of California 

Graduate School of Education 

Los Angeles, California 90024 

Dear Dr, Popham: 

Thank you for your letter of September 25 with the description of 
the U. S, Office of Edi cation supported training program being developed 
by you at UCLA. There is a genuine need for the work you and your group 
propose to undertake. 

There is definite interest in our using the services of such a train- 
ing program. We are concerned with product evaluation and educational 
development. From the brief descriptions of those two specialties, I feel 
that not only my central staff but my field staff would profit. This means 
about 175 people. 



Please keep me informed. 



incerely, 




RHS/lcs 



IB 



OREGON 

BOARD OF EDUCATION 




942 LANCASTER DRIVE NE • SALEM, OREGON • 97310 • Ph. (503) 364-2171 Ext. 1 60 



TOM Me • ll 

GOVERNUK 



October 8, 1970 



BOARD OF EDUCATION 



Cr. ElEANOR BEARD, Chairman 
1560 5. Sky I and Driv« 

Lake Oiwero 97034 

RICHARD F. DElCH, Vic« Chairman 
1010 COrbtlf Building 
Portland 97204 

EUGENE H. FISHER 

Kellogg Route, Box 91 
Oakland 97462 



FRANCIS i. SMITH 

600 Morgan Park Building 
Portland 97205 



w. warr:n maxwe !. 1 

Roufa i, Box 1 4a 
Lakevlew 97630 

FRANK J, VAN DYKE 
110 E, Sixth 
Medford 97501 



FRANK M. WARREN 
621 S.W. Alder 
Portland 97205 



DALE PARNELL 

Superintendent ind Executive 
Officer of the Board 

JESSE FASOLD 

Deputy Superintendent and 
Secretary of the Eosrd 



Dr. W. James Popham 

Pro f essor of Education 

C3nter for the Study of Evaluation 

U. C. L. A, Graduate School of Education 

Los Angeles, California 90024 

Dear Doctor Popham: 

Thank you for your letter telling of plans for a training project 
in the areas of educational development and educational product 
evaluation. Sounds great! 



We are interested in this kind of training, both areas, for a 
number of our staff members. If we are unable to provide "on 
the job" or in-service within our own staff, we will be looking 
for other sources. 



Cordially, 



ft (Suu 



Maurice D. Burchfield 
Director, General Education 



MDB: js 

cc: William Loomis 
Ray Osbum 
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THE AMERICAN COLLEGE OF LIFE UNDERWRITERS 




D+* -JV C LANGDON 

C'm-C tok of instructional des.gn 



November 10, 1970 



Dr. W, James Popham 
Professor of Education 
University of California 
Los Angeles, California 90024 

Dear Dr. Popham; 

Thank you for your letter of October 1 concerning your U.S.O.E. 
contract to prepare educational product developers and evaluators. 

Our Executive Director, Dr. Harold Rahmlow, in an October 15, 1970 
reply to you indicated that I would be contacting you relative to my 
own specific areas of interest and respcnsibili ties within the Adult 
Learning Laboratory. Knowing of your outstanding work in the area of 
instructional technology, it is indeed a pleasure to think of the pro- 
spects of our working directly with you. 

As Director of Instructional Design, my primary responsibility 
will be for the production-development end of our research and develop- 
ment activities. Our organization is essentially broken down into 
three broad areas under the Executive Director. These areas are: 

Research and Evaluation, Instructional Design, and Technical Systems. 

The first two may be self-explanatory. Technical Systems has as its 
thrust the specif ication, acquisition, installation, maintenance and, 
most importantly, nodifica t ion of hardware systems relative to the 
purposes of m> own area, Instructional Design. 

In terms of the Instructional Design aspect of our tripcd organiza- 
tion, possibly the enclosed systems design I have developed will best 
explain how we, in general, will approach the developmental process 
through ana ly s i s, des ign, va 1 ida tion , implementation and maintenance. I 
have prepared a guide following the U. S. Office of Education prtposal 
format for the submission of our own proposals as the initial stage in 
this total systems design. Having developed this systems design, we 
then proceeded to develop an organizational framework in order to carry 
through our activities, both on a research and development basis. I have 
already indicated the three broad areas of that organizational framework. 
Within my own area of Instructional Design, I will basically have a staff 
of writers and programmers working with subject natter experts on the 
one hand and also the production petple in terms of all forms of media 
under my control such that we can take a software -oriented approach. With 
the systems design and organizational framework in mind, I think you can 



o 
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easily see our commitment to Instructional Technology and thus the need 
for the finest in product developers and evaluators. Our evaluators and 
developers will have a strong relationship, however, in our research func- 
tions as they may well be serving both functions. 

As Dr. Rahmlow indicated, at the present time we have a very small 
staff. We aie planning programs and future activities of the lab. Since 
our physical facility will not be finished until the early part of 1972, 
we will probably not have until that time requirements for a substantial 
number of developers and evaluators. I definitely See us in a position 
of assisting in the training of both developers and evaluators, as well 
as some other technical types of people. I had in mind some time ago 
the possiblity of using some students from such places as Florida State 
under Dr. Robert Morgan, or students, let's say, from the University of 
Illinois or possibly Michigan. It sounds to me as if the Intent of your 
program would fit more closely into our needs, however. 

I think it would be advantageous at some point in the near future 
for you or your representative and our organization to exchange ideas 
on a direct basis. We usually manage to get to most of the major conven- 
tion meetings and, if aware of your plans for NSPI, AP\, DAVI, etc., pos- 
sibly we could arrange to get together. We at the Adult Learning Labora- 
tory are anxious to furt! er explore the implications of your program and 
the sharing of mutual research and development results. Please let me 
know what the next step is to ue . Thank you. 



Sincerely yours, 




Danny G. Langdon 



DGL/lmb 
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STATE OF MICHIGAN 



DEPARTMENT OF EDUCATION 







STATE BOARD OF EDUCATION 



Lansing, Michigan 48902 



PF.TER OPPEWALL 
President 

THOMAS J, BRENNAN 
V»c« President 

> ’CHATL J. |>EF.B 
Secretary 

JAMES F. O’NEIL 



October 19, 1979 



JOHN VV. PORTER 
Superintendent 
of Tublic Instruction 



GORTON RIETHMILLER 
MARILYN JEAN KELLY 
CHARLES E. MORTON 
EDWIN L. NOVAK, O D. 



Dr. w. James Popham 



CGV. WILLIAM G- MILLIKEN 
Ex-Officio 



Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 9002*1 

Dear Dr. Popham: 

Thank you very much for your letter describing the proposed 
training program in educational development and educational 
product evaluatior. The entire project sounds quite interesting. 

I am the director of a division, within the Michigan Depart- 
ment of Education whicn employs forty professional consultants 
in various categorically funded federal programs and some general 
subject area consultants. We do feel a need for training experiences 
in the areas you suggest. However, I would be unable to indicate 
a level of participation until I receive more specific infomation 
about the institute, the specific types of programs offered, costs, 
and of' particular import cnee , the length of time the various types 
of activities would require. 

Hie to budgetary limitations within o’ir Department ( a problem 
I am sure is shared by many state departments) we have to be some- 
what careful about the types of experiences our professional staff 
can participate in on an out-of-rtate basis. For this reason we are 
naturally interested in activities as close to Lansing as possible. 

Aliy farther infoination you can provide at sane future dale 
will be appreciated. 

I hope this infoimation is sufficient for your pm poses at the 
present time. 




Richard E, Barnhart 
Director 

CuiTiculum Division 



REihrs 
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DETROIT PUBLIC SCHOOLS 

DIVISION FOR ADMINISTRATION OF SCHOOL 

CONTINUING EDUCATION, EVENING AND 
SUMMER SCHOOLS 

5057 WOODWARD DETROIT, MICHIGAN 48202 PHONE 313/834-7900 

October 22, 1970 



Professor James W. Popham 
Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 

Dear Professor Popham: 

I am writing you on behalf of our division head, Dr. Charles E. Stewart, 
and the Office of Continuing Education of which the Department of Teacher 
Education is an integral part. We are very much interested in your train- 
ing programs to prepare educational product developers and educational 
product evaluators* Vfe, in the Office of Continuing Education, would like 
to send some of our teacher education specialists and possibly some of our 
school-based curricular leaders to training sessions of this kind, ibis, 
of course, depends upon the availability of funds and when these training 
sessions are to be held. 

Would you please keep us informed of the progress of your program design. 

We are especially interested in the shorter, specialized courses of two 
weeks duration. 

If you find that you would like to test your program design somewhere 
away from California’s sunny clime, we would like to explore that possi- 
bility with you. We have had some experience with consortiums . Between 
40 and 30 Detroit teachers and administrators are about to take part in a 
seven-day training session sponsored by the National Media Institute, it 
seems to augur well, and we are looking forward to it with great expecta- 
tions . 

Vfe ar^ pleased that you have informed us of your much-needed and innovative 
enterprise. Please keep us informed of your progress. 

Sincerely, 

Norman McRae 
Assistant Director 
Teacher Eiucation 





enclosures 



cc: Dr, C.E. Stewart 

Dr, H.a. Hart 
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BOARD OF EDUCATION; FETER F. QRYLLS JAMES A. HATHAWAY FAT 
REMUS O ROSINSON. M .D. FEV. OARNEAU STEWART 



ICK A. MCDONALD 
A. L. 2WEAOLIN« 



AlRDUl 



APPALACHIA 

Educational Laboratory/ Inc. 

P. 0. BOX 1348 

CHARLESTON 'VEST VIRGINIA 25325 



October 7, 1970 



Dr. W. James Popham 
Professor of Education 
University of California 
Los Angeles 

Center for the Study of Evaluation 
U.C.L.A. Graduate School of Education 
Los Angeles, California 90024 

Dear Dr. Popham: 

This is to respond to your letter of September 28 inquiring of our 
interest in assisting with training educational product developers and 
educational product evaluators. From sharing your request with the 
Laboratory Deputy Director, Dr. Robert Childers, he suggests that we 
respond with two important points: 

1. We have difficulty in finding trained personnel and 
therefore would be very eager to discuss employment 
with newly trained individuals in both specialities. 

2. There is a need among our present staff for short, 
intensive training sessions for developing a number 
of specialized skills. This might be accomplished 
through special workshops or sel f -instructional packages 
which your staff may produce . 

To provide you further information relative to the Laboratory, 3‘ am 
enclosing a section of our 1971 Contractor's Request for Continued Funding 
on "Research and Development Stages." The significance of this is to pro- 
vide you an abstract of the Model employed by the Laboratory in educational 
development. Additionally, I am enclosing an organization chart of the 
Laboratory . 



Dr. W. James Popham 
Page Two 
October 7, 1970 



We will appreciate hearing further from you. 

Sincerely yours. 




Director 



BEC/bj 
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KMC CORPORATION 
7000 SANTA MONICA 
BLVD * HOLL'i WOOD 
CALIKORNIA * 9003* 
HOllywood 3*3282 * 

October 7 , 1970 



W . James Popham 

Center for the Study of Evaluation 
UNIVERSITY OF CALIFORNIA, LOS ANGELES 
Graduate School of Education 
Los Angeles, California 90024 

Dr. P oph am : 

We are interested in your training program, specifically the shorter, 
specialized courses in educational development. There might also be 
a need for some of our people to further familiarize themselves with 
the processes of evaluating educational products. 

Our firm has been a producer of educational materials in a variety of 
audio-visual media for many years, and enjoys a reputation for quality 
to the standards of the most stringent educational evaluators. 

But we would always be interested in seeking further means of building 
the strength of our own developers and evaluators for a greater in- 
house capability in developing new programs. 

We are still considering educational programming as presented to us by 
independent producers. These outside sources of original program 
content are usually experienced educators. 

Towaid these needs, we would continue to be pleased to accept proposals 
from those associated with you, as we have often effected a mutually 
beneficial relationship with educators from colleges throughout the 
country . 



Meanwhile! I extend our very best wishes for the successful realization 
of all yqAir goals in the establishment of a new training institute. 



Sincerely 



Art Cc 

Project Director, 

FILM DESIGNERS DIVISION 





t DUCAT ION At MATtRIAI S DIVISION * l» VAST t\n STRUT * Sf. PATl . MINNESOTA 551 Cl 
AC/nn HI \1 SIGNERS DIVISION? * 7000 SANTA MONICA BLVO * HOI n WOOD o.W? * ^ * r 
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RADIO RECORDERS DIVISION v :000 SANTA MONICA BIVD * HOimvOOI) 




miiwaukee public schools 



DIVISION OF CURRICULUM 
AND INSTRUCTION 

administration building 
5225 west vllet st: p.o. draper 10k 
miiwaukee, Wisconsin 53201 
area 414:476 3670 



October 7, 1970 

Dr* James Po 

Cente" for the Study of Evaluation 
U.C.L-.A, Graduate School of Flucation 
Los Angeles, California 90024 

Dear Dr. Popham: 

Thank you for your letter of September 28 inquiring about our interest 
xr your training program for educational product developers and evaluators. 
Without knowing more specific details a9 to objectives, schedules, costs, 
etc, I woulf be hesitant to indicate more than interest. We do have 
several staff members interested in these areas who might be candidates 
for a training program such as you are planning. 

I'm sure that we would want to consider participation in the programs 
after details are available, especially in the shorter specialized courses. 

Please feel free to send more information as your plans develop, 

Sincerely, 




Norman M. Rose 

In-Service Education ft Volunteer Coordinator 
Division of Curriculum ft Instruction 



NKR/esg 
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COMMONWEALTH Of PUERTO JUCO 

DEPARTMENT OF EDUCATION 
HATO REY. PUERTO RICO 



OFFICE OF PLANNING 
AND EDUCATIONAL DEVELOPMENT 



October 2 J ] 9/0 



Mr. W. James Popham 
Professor Education 
University of California 
Los Angeles, California 

Dear Mr. Popham; 



Your letter addressed to Mr. Plaud regarding the 



training program being designed at UCLA, was referred to us. 
Mr. Pi aud is in the States for a week or so. 



Although an official answer to your inquiry regarding 



training requirements in the two indicated fields will eventually 
be forwarded to you by Mr. Plaud , I wish to anticipate that we are 
very much interested and much in need for the type of training your 
letter describes. If the training program is set up for the 
summer 1971, it seems we shall be in a position to arrange for parti- 
cipation. 

L r e would need further information regarding cos t , dates, 
number of participants allowed, etc. * 



Although we are in a position to inform you some of 



our training needs I would prefer to include ours iwth the complete 
listing that Mr. Plaud and other key persons in the Department can 
provide. I shall do my best to assure this information reach you 
in the nearest future. 



Sincerely 




Direc tor 

Evaluation Office 
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JAM HANDY PIIODUCTIONS 

nivisinN t t ~ no. 

282 1 EAST GRAND BOULEVARD ■ DETROIT. MICHIGAN 482 1 1 
AREA CODE 3 1 3 . 8 75-2450 



October 22, 1970 



Prof. W. James Popham 
University of California 
Los Angeles, California 90024 

Dear Professor Popham: 

I am replying to your recent letter to Norm Stanton 
as he is away on vacation. 

Several of us here who are deeply involved in the 
creative aspect of preparing training programs for 
our large industrial clients were both surprised and 
tremendously pleased to hear of the work your group 
is doing in educational product development and 
evaluation. Certainly there has been a great void 
in this area, and we have hoped for many years that 
effective work would be undertaken at the University 
level . 

Based on the information you have made available, 
we feel that the training institute you envision 
will be a very worthwhile activity, and one that will 
certainly warrant our support. Obviously, it is dif - 
ficult to estimate the use we might make of this nev 
resource until it is possible to see further details 
as to the nature of the training to be offered. If 
your work does correspond to our needs in business 
communication and training (particularly in marketing 
and management areas), we will certainly be interested 
in employing institute graduates from time to time, 
and will almost assuredly want to send a few of our 
people for training in valuation. 

I might suggest that the critical element in deter- 
mining our interest would be the degree to which your 
work will lend itself to the design of highly parti- 
cipative education tending to follow the conference, 



0 
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PROGRAMS ANO VFDIA FOR BUSINESS COM MU NIC AtlONS j ElLM . VlHEO 1 Art • PACK*C,tO Mt C TIN 



•ini 



Prof* Wo James Popham 



Page 2 



.mi* 




workshop, case history methods. While we do produce 
and utilize numerous educational media items in the 
visual area, we do very little work with text books 
or typical classroom lecture situations* 

We certainly applaud your efforts, and will be looning 
forward to hearing more from you as the program moves 
ahead* 



Sincerely yours, 




HH: jm 



cc: Mr, Norman B* Stanton 
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SAN DIEGO CITY SCHOOLS 



EDUCATION CENTER 

AND PL CAJJN BOULEVARDS 



October 8, 1970 



4 too normal street 
san Dr eg o , California *2103 



CU RRICU LU M SERVICES DIVISION 



Dr. N. James Popham 

Professor of Education 

Center for the Study of Evaluation 

University of California 

Los Angeles, California 90024 

Dear Dr. Popham: 

This is to indicate a definite interest on the part of the San Diego 
City Schools in the proposed training program for product developers 
and evaluators. 

The San Diego City Schools has a substantial summer program in 
curriculum development and year around program of in-service education. 
We feel that product de\eloptnent and evaluation will be a significant 
addition to our program. 

The San Diego City Schools also volunteers, on the basis of its 
experience, any assistance needed in the development of the proposal 
or the carrying out of the project. Please let us know if we can be 
of any assistance. 

Sincerely, 




WHS :dg 



ERIC 
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Southeastern education Laboratory 



Dr. W. James Popham 

Professor of Education 

Center for the Study o 2 Evaluation 

UCLA Graduate School of Education 

University of California 

Los Angeles, California 90024 

Dear Dr. Popham; 

Thank you for your informative letter dated September 28, 1970, 

briefly outlining the plans for a training program to prepare educa- 
tional product developers and educational product evaluators. 

The Southeastern Education Laboratory would be interested in 
having approximately six of our staff members participate in such 
a course in educational development during 1971, and approximately 
three of our staff members participate in such a course in educa- 
tional product evaluation during 1971. The Lab would also be inter- 
ested in employing one or two new staff members each year who have 
been trained by your proposed year-long basic training program in 
educational product development and educational product evaluation. 

Please keep us informed of your activities as you continue to 
design and operate your new program. 



3450 INTERNATIONAL BOULEVARD, SUITE 221 
ATLANTA, GEORGIA 30354 



KEIOi’E r H W. T1DW2LL 

Executive Director 



October 7, 1970 



TELEPHONE! ( 404 ) 760*09 5 1 



Sincerely your& 




Kenneth W. Tidwell 
Executive Director 



KWT : BLH 
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ATLANTA PUBLIC SCHOOLS 

INSTRUCTIONAL RERVJCR0 CENTER I9SO FORRE1T HILL DRIVE. S,W, ATLANTA. GEORGIA 303 IB 



OFFICf OF 

AMIRAN! ■ UPBRINTtNOINT 

roRiN-TRucrioN October 6, 1970 



Mr. V. James Popham 
Professor of Education 
University of California 
Los Angeles, California 

Dear Mr. Popham: 

rhank you for your informative letter regarding the program 
you are designing. 

The Atlanta School System would definitely be interested in 
utilizing the services which you have described, particularly 
those relating to the training in educational product development 
and evaluation for members of our staff. 

Please keep us apprised of your progress as the program 
develops . 

Sincerely, 

S/' Sl'l 2 ^ ^ 

' R. Ruel Morrison, Director 
Teacher Education 

RRM : vm 
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COMMONWEALTH OF KENTUCKY 



'Separinmtt of ^fbucaium 

FRANKFORT 40601 



October 5, 1970 



Dr. W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 

Dear Dr. Popham: 

I have discussed your letter of recent date with the 
Director of our Division of Research and Comprehensive 
Planning and he feels that your proposed program has 
great possibilities in the training of personnel in 
the specialities of (1) educational development and 
(2) educational product evaluation. I might indicate 
that we have no current training requirements in these 
two fields of emphases. 

We are very interested in your planned program and 
would like to have additional information as it becomes 
available . 

Yours very truly, 

Richard L. Winebarger 

Coordinator 

Title III, ESEA 

RLW : rdw 
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STATE OF SOUTH DAKOTA 
DEPARTMENT OF PUBLIC INSTRUCTION 




Pierre 57 501 






DR. GORDON A. DltOT RICK 

ir^7£ SUPERINTENDENT 

October 5, 1970 



AC 605 224-3011 



ELDON E. GRAN, Ed. D. 
/JssisfarU Superintendent 
Instructional Services 



Dr. W. James Popham 
Professor of Education 

University of California* Los Angeles 
Los Angeles* California 9002*f 

Dear Dr. Popham: 

In answer to your letter of September 25, our state 
agency would very likely wish to send staff members for- 
training, expecially in the area of educational development. 
Our work lies chiefly in identifying curriculum needs and 
helping schools to devise materials and procedures to meet 
them. I would suspect our staff would be more interested 
in short-term courses based on highly specific objectives, 
unless the long-term leads toward the doctorate. We have 
two staff members who are eligible for sabbaticals; both 
wish to pursue a higher degree. 



Sin cere ly , 








Eldon E. Gran, Ed.D. 
Assistant Superintendent 
Instructional Services 



EEG : b j 
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DEPARTMENT OF THE ARMY 

UNITED STATES ARMY INFANTRY SCHOOL 
FORT 0ENNING. GEORGIA 31 905 




IN REPLY REFER TO 



ATSlN-H 



5 October 1970 



Dr. W. James Popham 

University of California 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

*-05 Hilgard Avenue 

Los Angeles, California 90024 



Dear Dr. Popham: 

Thank you for your recent letter concerning your efforts in i : ■ 
a new training program for educational product developers eu-. ’ d 
tional product evaluators. From the description of your pr< i 
would appear that the United States Army Infantry School wo. u ■ 

to utilize the services of such training activities. Of a necef 
availability of the program in relation to the availability of fu 
appropriate personnel here at the Infantry School, at the tim< 
course, would have a direct bearing on our actual participaUu 

For over two years the Infantry School has been, and cur re n / 
deeply involved in the systems engineering of its courses of i , i m . 

The course design centers around developing <y stematically a p am 
of instruction based on duties, tasks, and performance obje a H 
proper sequencing of material; the development of standard . cc 
of performance and learning; appropriate cri d^ria for evaluau 
selection of the appropriate media and presentation technique 
throughout the process, the proper and effective quality coni ■ 
instructional system. 

The Infantry School uses, in addition to the rather conventii 
of instruction (such as the lecture, conference, and emonsli.i 
specialized techniques and systems. Three of the 200-man 1 

in Infantry Hall (our main academic building) are equipped v. i u 
response systems (EDEX). One of the Infantry Hall's 200 «m 
rooms has 50 cathode ray tubes for the presentation of CAI 
almost all of the classrooms in the main school complex ar< 
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ATSIN-H 

Dr, W. James Popham 



5 October 1970 



television receivers which are serviced by a TV facility (two studios 
and a mobile facility), providing for live productions, development of 
in-house TV tapes and the use of video tape playback (including film 
chain) capability. Attached brochures (Inclosure 1} give some over- 
views of some of these systems. 

Three of the Infantry SchooDs main courses; namely, the Infantry 
Officer Advanced Course, the Infantry Officer Basic Course, and the 
Infantry Officer Candidate Course, are structured so that the student 
has the opportunity to take electives - -some of which are presented via 
the Georgia State University System, others by independent study, and 
still others via in-house seminars. Other courses are primarily skill 
type courses. Efforts are being made to move more into individualized 
instruction throughout all courses. Plans are under way for the develop- 
ment of a learning center and the expansion of some of the programmed 
instruction material. 

To give you a picture of the framework of the Infantry School, the fol- 
lowing are some basic facts; The Infantry School has an annual input 
of approximately 50, 000 students per year. The average daily enroll- 
ment in 1970 was approximately 10, 000; in 1 971 , it will be about 7,000. 
There are 25 distinct courses varying in length from 1 week to 36 weeks; 
225 classes attend these courses. The level of students is from privates 
through general officers. Some of the courses are skill courses; some 
are broad educational courses. The staff and faculty is approximately 
3,300. Over 99% of the instructional staff are military. Instructor 
personnel, noncommissioned officers, as well as officers, all undergo 
an intensive three-week Instructor Training Course. 

I trust that the above discussion gives you some ideas of the Infantry 
SchooDs instructional program and its involvement in educational 
product development and educational product evaluation. Since most 
of the instructional personnel, being military, have had little formal 
instruction in the field of education, we must train these individuals in 
our Instructor Training Course and in our Programmed Instruction 
Workshop. There is a need fox the personnel involved in this training 
to have a sound basis in the areas you discussed in your letter. Some 
of the civilian education specialists concerned with instructor training 
and program development do have a sound background; others need 
some additional training. Accordingly, there would appear to be a 
need for some of the military instructional staff and civilian education 
specialists to participate in training such as I envision you are attempting 
to develop. 

I would appreciate your keeping me informed of your progress and plans. 
Please keep in contact with me. You may wish to contact me via phone. 



2 



atsin-h 

Dr. W. James Popham 
I may be reached via the following: 



5 October 1 970 



Richard S. Kneisel 

Special Assistant to the Commandant - 
Education Advisor 
US Army Infantry School 
Fort Benning, Georgia 31905 



Phone: Area Code (404), Extension: 545-1332 or 545-2021. 



Sincerely yours, 



1 Incl 
as stated 




Special Assistant - 
Educational Advisor 
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ERJC 




commonwealth- of Virgin 




State Department of Education 



RICHMOND. 23116 

October 6, 1970 



Dr. W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 90024 

Dear Dr. Pophara: 

Your recent letter outlining tentative plans for a training program to be offered 
by a consortium sounds interesting. 

When your plans have firmed up we would be happy to have more inirrmation. We 
are cognizant of the need to provide training in product evaluation for State staff 
members as well as staff of LEAs , It is our intention to avail ourselves, of com- 
petent training in evaluation techniques. It may be that your organization will be 
able to meet our needs. Please keep us informed. 



Sincerely , 



(Miss) Anne E. Tucker 

Supervisor, ESEA Title III 
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HUBERT WHEELER 
COMM 3510*" 



Area Cooe 3 1 4 
Pho*« 635-8125 




STATE DEPARTMENT OF EDUCATION 



DIVISION OF PUBLIC SCHOOLS 



JEFFERSON BUILDING 
P. 0. BOX 4 SO 



JEFFERSON CITY, MISSOURI 65101 



October 2, 1970 



0 

ERLC 



Dr. W. James Popham 
Professor of Education 
University of California 
Los Angeles, California 90024 

Dear Jim: 

Reference your letter concerning the product development/ 
evaluation training sessions which are now in the process of 
formulating, I discussed our interest in this with the people 
in our division. They indicate that we definitely have an 
interest in this area but before we can make a firm commitment 
we will have to know tore about cost, released time that would 
be necessary, etc. I realize that at this time you are probably 
not ready to provide that information bur if at some future 
date you could provide it to us, I might be able to give you a 
more definite answer. 

At this stage of the game, I think I could say, with reasonable 
certainty, that we are much interested but not able yet to 
commit. 



Sincerely yours 



Do.. ~ . 

Di rector 
Title III, ESEA 
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Oivision cr 

Cur^iCuvum and instruction 



September 29, 1970 



W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 90024 

Dear Jim: 

I wanted to take this opportunity to inform you that Jim Hinkle has 
resigned as Title III-ESEA Coordinator to assume the position of Administrative 
Assistant to Aaron Fink, Superintendent of Newton Public Schouls. I have 
assured Jim's position within the Department of Education effective as of 
September 22, 1970. 

In addition to answering your letter, I also received ION material last 
Friday, I am impressed at the quantity of the material as well rs the quality 
from quick perusal. However, I have not had an opportunity to make a thorough 
evaluation of all this in formation . 

As for the training program you envision in the summer, it sounds very 
exciting. At this time we are in the mist of acquiring new staff and I am 
not at all sure of haw the training program oould be utilized. Hcwever, I 
am excited at the concent of such a training program and am sure that I and 
mv staff oould utilize such traininq in the future. 

A quick question in reference to the second segment of the program, Educational 
Product Evaluation. Do you envision the establishment of a Curriculum Analysis 
Materials System similar to the one devised by Bill Stevens and Ervinq Morrissett 
at the Social Science Consortium in Colorado? 

Please keep in contact in the future. If I can be of any assistance to 
you, please let me know. 



Sincere ly , 




Don Torres 

KSEA Title III Coordinator 
Bureau of Curriculum InnovaKon 
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STATE OF WASHINGTON 



jiupcruitrnfrent of public ^lushructunt 



LOUiS BRUNO 



p- O BOX 327 
OLYMPIA 90301 



STATE SUPERINTENDENT 



September 28, 1970 



Dr. W. James Popham 

Professor of Education 

University of California, Los Angeles 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Ango 1 r , California 90024 

Dear Dr. Popham: 

Thank you for your recent letter describing a training program which you 
are presently designing under a contract from the Office of Education. 

We are extremely interested in both the product development and product 
evaluation components of this program. We aie also interested in partici- 
pating actively in them. 

It Is difficult to project the degree to which we could be Involved as 
this would depend on certain administrative decisions occurring at another 
level of this agency. 

Be assured, however, that we are vitally concerned with your plans and 
hope that you will keep us informed as you move forward. 



Best regards 




DIVISION OF CURRICULUM 
AND INSTRUCTION 



Rich Boyd 
Administrator of 
Title III, ESEA 



RB: ie 
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cc: Dr. Donald Hair 



STATE BOARD OF EDUCATION 



STATE BOARD OF EDUCATION 



ALLEN P, BURKHAROT, P«SI0E!*T 
P o. sox I 74 
NORFOLK 66701 



ROBERT G. SIMMONS. JR., VICE-PREUDINT 
2121 FIFTH WEN U E 
SCOTT SBL UFF 69361 



jiiate of Nebraska 
Jepartnmtt of Jlwcatimt 



LLOYO V. WRIGHT 
BOX 76 

RET f;OLD 9 6 6 4 2 9 



NORMAN OTTO 



920 EAST ANNE 
MILLARD 66043 



FRANK E. LA NO 1 S 



JOHN K. LONOAY 
4326 HARRISON 
OMAHA 66117 



*619 VAN DORN 
LINCOLN 66502 



CECIL E, STANLEY 



commissioner of eoucation 



JOHN A WAGONER 
17 10 HARRISON ROAD 
GRAND ISLAND 66801 



STAT* CAPITOL ; > . LINCOLN *6509 



O E BORA H H SHEPHERO 

ROUTE 1, BOX 22 
LEXINGTON 68650 



September 28, 1970 



Dr, W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 90024 

Dear Jim: 

In response to your letter advising us of the 
training program you are preparing, the Nebraska ESEA 
Title III Office would be interested in both the Edu- 
cational Development and Educational Product Evaluation 
training programs which you are proposing, 

I feel that we could use these resources not only 
to provide training for members of our staff, but pos- 
sibly in the future, to employ individual: both at 
the State level and at the project level wno have re- 
ceived these trainings. 

Would you please keep me informed of the status 
of these programs so that if you do receive approval, 
we might be able to actively participate in the ac- 
tivities . 




Jack Bail lie 
Administrative Director 
ESEA Title III 
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BANKof AMERICA 



SAN FRANCISCO HEADQUARTERS 



R. F. CHRISTIE 
Vice President 



September 28, 1970 



Mr. W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 90024 

Dear Mr. Popham: 

Your letter addressed to W. T J. Robertson regarding training 
analyses and development has been Deferred to me for reply. 

I am indejd interested in your oroject of developing train- 
ing courses on training analysis, program development, and training 
evaluation. This is much needed. There are too few trainers who 
understand this process, which, in my estimation, is essential if 
one is to make a contribution to his organization. 

If you have any additional information I would appreciate 
your sending it to me. Meanwhile, 1 w 7 'll anxiously await the 
development of your program and your reply. 

Very truly yours 





\2 



BANK OF AMER'CA NATIONAL TRUST AND SAVINGS ASSOC ATlON • BANK OF AMERICA CENTER . SAN FRANCISCO. CAl FORN'A 94i20 



DIMENSION FILMS 

733 NORTH LA BREA TEL WEBBTER 73300 
LOB ANGELES 38 CALIFORNIA 



September 25, 1970 



Dr. W. James Popham 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate 5chool of Education 

Los Angeles, California 90024 

Dear Dr. Popham; 

I was most interested to learn about the Center for 
the Study of Evaluation in your letter of September 17. 

Our company specializes in the development and 
production of educational films and supporting printed 
materials. We have been following with keen interest 
the trend toward accountability. Several years ago one 
of our school advisers stated that even films such as 
ours, which are primarily aimed at the affective domain, 
would sooner or later be subject to accountability. In 
fact he believed that the company which would develop 
its own measuring device and publish the results along 
with its new products would gain a considerable 
advantage over its competitors. 

We have, up to now, not made any such effort. Partly, 
we share the general dislike and apprehension of most 
practitioners and craftsmen who are asked to account 
mathematically for a process they have generally 
pursued intuitively. Also, we have been very skeptical 
of the possibility of measuring with any accuracy the 
complex subtle attitudinal results of viewing good 
motion pictures. We, for example, read with a combina- 
tion of amazement, good humor and horror the reports of 
research in Audiovisual Comrtunica tions Review. 

However, we recently received from the ASCD a booklet 
entitled "Improving Educational Assessment and 
Inventory of Affective Behavior." I also recently 
learned of the studies being conducted at the Stanford 
Research Institute, the University of Michigan, and the 
Russell Sage Foundation to develop master social 
indicators, including measures of social psychological 
attitudes. So apparently, whether we like it or not, 
the scientists are catching up with us. 
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Dr. W. James Popham 



On the general principle that it's better to join 
than fight, we would be interested in cooperating in 
any experimental efforts to develop evaluation 
methods for such educational goals as heightened 
sensory awareness, improved decision making in 
situations of value conflict, more effective 
response to personal stress, and greater openness to 
communicate personal feelings. These are all goals 
of some of our recent and current films; and if you 
can help us to determine how effective these films 
are, we would be most grateful. 

You ask about our possible use of graduates in the 
future. We expect to continue doing about the same 
kind of thing for the next few years so that even if 
you can't help us now, we would certainly consider 
working with one of your graduates at a later time. 
Since we are a small company, this would not be a 
staff position but more likely an agreement for cne 
job at a time. 
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CENTRAL INTELLIGENCE AGENCY 

Washington, D.C. 20505 



29 October 1970 



Dr. Vi. Janes Fopham 

Professor of Education 

University of California, Los Angeles 

Lob Angeles, California 90024 

Dear Dr. Pophami 

The training program you are developing to prepare educational 
product developers and educational product evaluators would be of 
interest to ua if it lends itself to skills training. We are 
involved in determining instructional objectives and evaluating 
training, with the emphasis on the student's ability to perform 
a particular skill at the conclusion of the training. 

I can't say at this time that we would be able to send any 
students to the program, but ve would like* to be kept informed 
of its development. I would appreciate it if you will let me 
know the dates and tuition costs of the courses when they become 
available. 

Ve wish you success in your new project and look forward to 
hearing more about it. 



31 ncerely, 

/ 

C 1 /;< t// — 

Hugh Clayton / 

Chief 

Inst ^ict ional Support Staff 
Oiitce of Training 



State of Ohio 

Department of Education 

Columbus 



D I u I i r s OF 

GUIDANCE AND TESTING 




M w. E^SE' 
SUPERINTENDENT OF 
F 0 BL I C IN STRUCT I ON 



751 northwest B L V D . 
COLUMBUS, OHIO 43212 
PHONE 469-4590 



November 2, x 970 



JOHN G . ODGERS 
M B F C t Of 



Dr. W. James Popham 
Center For The Study Of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 



Dear Dr. Popham; 

From the information in your recent letter, it appears that your project 
designed to train educational product developers and educational product eva- 
luators may be of interest to several of the Divisions of the Ohio Department 
of Education. Some of the ideas seem to fit in with the objectives of a number 
of projects that we have underway. The short specialized courses sound es- 
pecially interesting. 

The Ohio Department of Education, like many other state educational 
agencies, has almost no funds available for sending personnel to the types of 
programs that you are planning. If it is possible, I recommend that your 
program proposal to the U.S.O.E. include a provision for funding the partici- 
pation of state educational agency employees in your programs. 

Please keep me informed of the progress of your project. 

Sincerely, 

-xl.v 0 Atf 

Frink O'Dell 

Induction Training Project Coordinator 

FLO: be 

cc: John G . Odqers 
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STATE OF HAWAII 



DEPARTMENT OF EDUCATION 



HONOLULU 



HAWAII ENGLISH PROJECT 
1 6 2 5 WIST PLACE 
HONOLULU, HAWAII 96622 



October 14, 1970 



Mr. W. James Popbam 

Professor of Education 

Center for the Study of Evaluation 

UCLA Graduate School of Education 

Los Angeles, California 90024 

Dear Mr. Popham: 

Your letter to Dr. Arthur Mann relative to your training programs in 
education product development and evaluation of educational products 
was referred to me. 

My Branch is charged with the responsibility of undertaking major 
curriculum revision, evaluation, and materials development projects. 
We currently have a staff which is undertaking the production of 
materials for a completely revised curriculum in the language arts-- 
grades K-6. My staff and I learn the skills of education product 
development largely through experience. None of us has received 
formal training in these skills. What you describe in your letter 
is, therefore, of considerable interest to us as a staff development 
activity. 

At this time what would be practical in the way of commitment to a 
training program would be an intensive training of a limited number 
of people (12 to 15) of key members of the development staff. The 
training period would be no more than 5 days. If such a training 
program could be organized for the summer months and .conducted within 
a budget that we could manage, I believe that we would be interested. 

I would appreciate hearing further from you on this matter. 




Si ncerely 



.'/S' - ; 1 t Aft 2 c' 1 

Sninkicnl Shimabukuro, Director 
Curriculum Development & Technology 



A TRAINING PROGRAM FOR 

DEVELOPERS AND EVALUATORS OF EDUCATIONAL PRODUCTS 



University 



of California 



Los Angeles 








December, 1970 

1 he products produced herein were performed pursuant to a contract with 
the Department of Health, Education, and Welfare, l\S. Office of Education 
under the Elementary and Secondary Education Act of 19f>V Public Law 89-10, 
Title IV. 
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This document was prepared and reviewed by 
many individuals. During the early discussions 
that ultimately led to the training program design 
described herein, Professors Marvin C. Alkin, Eva 
L. Baker, Evan R. Keislar, John D. McNeil, and W. 
James Popham all shared in contributing to the 
conceptual framework. Later, individuals repre- 
senting the several consortium agencies involved 
in the training program contributed both conceptual- 
ly and by preparing sections of the document deal- 
ing specifically with their organization’s training 
role and staff. A significant section of the docu- 
ment dealing with instructional objectives and mea- 
surement procedures was initially drafted by Dr. 
Kcislar. Dr. Baker contributed the bulk of the 

product development objectives/mcasuremcnt pro- 
cedures as well as several other key sections. The 
remainder of the document, including its final co- 
ordination, was the responsibility of Dr. Popham. 
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INTRODUCTION 



The proposed training program to be described in the following pages emerges from 
a common conviction of the numerous individuals who designed the program, namely, 
that the availability of high quality instructional materials can significantly enhance the 
effectiveness of our educational system. For the fairminded educator recognizes that 
our educational system is far less effective than it should be. The difficulty lies not in 
identifying the problem, but in devising schemes for wholesale amelioration. While 
increasing the availability of high quality instructional materials is not the only way to 
bring about improvements in education, it seems to offer certain advantages less readily 
identified in many alternative approaches. 

Only recently have educational leaders in this country urged the federal government 
to provide large scale financial support for the systematic development of instructional 
materials. With the establishment of (1) university-based research and development 
centers and (2) regional educational laboratories, the U.S. Office of Education made a 
major commitment to support the preparation of carefully developed educational products. 
The potential influence of educational products has been identified by Cronbach and 
Suppes (1969): 

A new product can have great impact. An intellectual conclusion on 
education has no effect on the classroom until teachers have accepted the 
idea, have decided to use it, and have made thousands of detailed deci- 
sions about its day-to-day application. A new product can put the con- 
clusion into practice with much less demand on teachers for comprehen- 
sion and detailed planning, as it lays out a plan that less imaginative 
and less self-reliant teachers can follow. If the teachers accept the 
scheme, a certain minimum level of excellence in the program is thus 
almost assured, provided, that is, that the product itself has been 
thoroughly engineered. 

Similarly, Glaser (1966) holds great hopes for the benefits to be derived from major 
efforts to develop educational products and to evaluate them rigorously: 

At the present time, the greatest potential for change seems to be in 
the redesign o! educational procedures and 'tools.' The materials and 
equipment with which teachers are provided are potent means for in- 
fluencing both teachers and pupils behavior. Less direct and non- 
specific attempts such as changing teaching behavior per se without 
providing appropriate materials will be less successful. 

The tremendous advantage of educational development as a vehicle for promoting 
Improvements in education is that inherent In such development is the possibility of 
multiplying proven instructional advances. The products yielded by educational develop- 
ment must be reproducible, that is, capable of being implemented in essentially the 
same fashion in other educational situations. This means that these products can be 
disseminated widely, hence used with literally millions of children. As indicated, this 
type of dramatic across-the-board improvement is desperately required by our educa- 
tional system at this moment. 

1 rom an economic vantage point alone, such development activities would obviously 
benefit the nation's schools. Almost a decade ago, Lumsdaine (1961) observed that the 
cost of developing instructional materials was well worth the amount of research and 
development devoted to perfecting them since: 

A given instructional instrument may be used for only a few minutes 
of any student's time, but the cost of perfecting it can be pro-rated in 
terms of a denominator representing thousands of students for whom 
the perfected instrument C3n be used. 
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Although the federal government has begun to spend funds in support of development, 
those involved in the enterprise recognize a grave barrier to the proper utilization of 
these funds. We do not yet have sufficient numbers of competent personnel to engage in 
development. Historically, there have been no long-standing training programs designed 
to produce either educational developers or those who can evaluate already-developed 
instructional materials. Although there are a few isolated training efforts in recent years, 
no large-scale programs are currently operating at a level needed to prepare the many 
competent product developers who are now needed and will be needed in years to come. 
In surveying the history of education in our country, there are numerous instances where- 
in major projects met with failure simply because those carrying them out did not 
possess the required expertise. It is proposed, therefore, that a major new training 
enterprise be immediately assembled to prepare the individuals who must be instru- 
mental in a major development-based attempt to promote striking improvements in 
American education. 

The focus of the training program design to be discussed in the remainder of this 
document will be exclusively upon the training of (?' educational developers and (2) indi- 
viduals who are skilled in the evaluation of « ready developed products. By focusing 
upon the training of outstandingly competent product developers and product evaluators, 
the agencies collaborating to design this training program believe they can best con- 
tribute to the improvement of our schools. 



Product Developers orid Product Evaluators 

Although to individuals who have been working actively in the development field the 
phrases Product Develop er and Pr oduct Hvaluator satisfactorily describe two specific 
types of educational practitioners, it is undoubtedly desirable to define these two spe- 
cialties with more precision. 

By product developers we refer to those persons who design, construct, revise and 
refine the numerous sorts of instructional materials employed by educators. Instruc- 
tional materials include filmstrips, films, audiotapes, videotapes, programmed texts, 
non -programmed texts, etc. In addition, instructional materials would include any sets 
of highly prescriptive directions or guidelines designed for teachers with a view to 
producing more homogeneous teacher behavior, for example, a detailed lesson plan calling 
for particular teacher actions at particular points during an instructional sequence. 
The essential ingredient in these instructional materials is that they permit an essentially 
reproducible instructional sequence to occur. Such essentially similar instructional 
sequences can be contrasted with the highly variable instructional activities which 
emerge in the classrooms of different teachers, each designing and carrying out an 
almost unique instructional enterprise. Product developers prepare reproducible mater- 
ials. 



The type of developer we are talking about is not the person who has historically 
generated instructional materials. It is not the individual who, with more or less in- 
genuity, has written a textbook or produced an educational film. We are concerned with 
developers who are educational researchers, that is, who approach the task of product 
development with the anticipation that their efforts will be guided not by intuitive judg- 
Plgnts regarding the product's quality, but by the performance of learners who have used 
the product. Cronbach and Suppes (1969) emphasize this difference nicely: 

There is an important distinction between mere developmental acti- 
vities and development (product) research. Design and production can 
be carried out with no systematic, disciplined inquiry. Indeed, in the 
course of educational history, most curricula, teaching materials, 
building designs, etc., have been brought to final form through no more 
than casual tryout. Until relatively recently this lack of rigor was true 
of invention and design in all fields of human endeavor. But one field 
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after another - navigation, agriculture, manufacturing, nutrition - has 
taken the forward step from folklore to casual empiricism to tech- 
nology. Controlled measurement and observation have refined products 
and procedures, with correspondingly better results. Education is only 
beginning to emerge into a technological phase. 

The power that facts on performance have to guide improvement is 
perhaps most concretely illustrated by the methods used in preparing 
"programmed" textbooks. These are designed so that the pupil writes 
one answer after another to questions on successive elements in an 
explanation. According to the theory by which most programmers 
operate, if he makes an error the explanation was unclear in some 
way, or moved too fast. Classroom trials of drafts are essential to 
make the text effective. Any spot where errors pile up is a spot to be 
revised in the next draft; often the nature of the errors show just what 
is the source of confusion. Similar but less formal micro-evaluation 
can be made of any instructional material in draft form. 

As we shall see later in this training program description, there are many points 
during the development of instructional materials where the product developer may 
operate as a specialist rather than as the compleat developer. For instance, certain 
product developers may specialize in the revision of instructional materials based on 
field trial data, others may emphasize the design of early product prototypes. Whether 
specialists or generalists, the educational product developers to be prepared by the 
proposed training program will be responsible for the production of those instructional 
materials which permit the conduct of essentially reproducible instructional sequences. 

Some (Gideonse, 1970) would define the role of the developer more broadly, iden- 
tifying as products to be prepare! not only what we have referred to as a reproducible 
instructional sequence but also almost any type of tangible artifact having to do with the 
schools, e.g., architectural plans for school buildings, tests, schemes for disseminating 
new ideas, etc. We are not quarreling with such conceptions of development; we are only 
using a restricted definition where the focus is admittedly on the preparation of the 
kinds of instructional artifacts previously identified. 

Turning to product evaluators, we are referring to those people who undertake the 
appraisal of already completed educational products. The "already completed" phrase is 
critical in defining the product evaluator's role, for it is apparent that during the course 
of preparing instructional materials the product developer engages in frequent evaluations 
of his not -yet-finlshed-product. As pointer out above, in-process evaluation based on 
learner performance is the key to the kind of product development we have in mind. But 
there are many situations in which judgments must be reached about extant products, 
irrespective of the procedures which were used to develop them. 

For example, schools are often forced to select among competing instructional 
materials, perhaps none of which were developed according to the schemes recommended 
by performance-based developers. The decision still has to be made. Which materials 
should be chosen? The product evaluator we wish to train will be able to assist those 
who must decide whether to adopt, retain, or discard already completed instructional 
materials. Once more, the kinds of instructional materials under consideration are all 
those mentioned previously, the only criterion being, as before, that they yield an essen- 
tially reproducible instructional sequence. 

Having defined the two types of educational specialists to be prepared by the proposed 
training program, we can see how these two compatible specialities can be coordinated 
in ? highly efficient program. There are many points where the competencies to be 
pronoted for the product developer will be idenrica 1 or at least comparable to those 
sought for the product evaluator. We will capitalize on these similarities, both instruc- 
tionallv and economically, since we ire preparing developers .and one type of evaluators 



o 

ERIC 



3 



53 



(not the whole gamut of evaluation specialists), there will obviously be basic differences 
in the training programs. But by focusing on this single kind of evaluation speciality, 
we can coordinate our two main training enterprises with both curricular and financial 
dividends. 



The Current Demonds for Product Developers ord Evaluators 

A recent investigation by the AERA Task Force on the Training of Educational 
Researchers (Sanders and Worthen, 1970) analyzed employers' perceptions of research- 
related competencies and shortages of personnel possessing such competencies. Inter- 
views were held with 58 persons who either employed or supervised research or research- 
related personnel in numerous institutional settings. Forty-two interviewees listed the 
function of development as absolutely necessary in their programs. When respondents 
were asked to rank various research and research-related functions in order of their 
importance for achieving the institution's program goals, development was ranked first 
by representatives of regional laboratories, research and development centers, and federal 
agencies. Although it is difficult to sort out the need for product evaluators from the 
data of this AERA study, the closest descriptive function ranked was ’’output evaluation” 
which includes judging the worth of educational products. Outcome evaluation functions 
were ranked as very important by school districts, state education departments, inde- 
pendent research organizations, and federal agencies. 

Two further studies by the AERA Task Force make use of the 1968 and 1969 
AERA employment service data to compare competencies reported by applicants and 
competencies required for positions listed. Among the conclusions drawn from the 1968 
data (Worthen and Sanders, 1970) was the following regarding the need for training 
evaluators (including product evaluators): 

There were more than three vacancies in evaluation for every appli- 
cant prepared in this area. It appears that the evaluation mandates of 
Titles I and III of the Elementary and Secondary Education Act of 1965 
and an increasing trend toward accountability may have had an effect 
in 1968 and necessitated more attention to training in this area. 

A similar analysis based on 1969 AERA (Oldefendt and Worthen, 1970) employment 
service data reflected enough discrepancies between applicants and vacancy descriptions 
to suggest possible trends. Specifically, "the number of vacancies in development in- 
creased in 1969 . . . Regarding evaluation, the needs were increasing. "In 1968 there 
were three openings for every applicant with skill in evaluation. In 1969 this discrepancy 
rose to almost four to one." 

Additiona 1 evidence of the manpower shortage is provided by Francis Chase's (1968) 
report regarding the difficulties that educational laboratories encountered in securing 
trained personnel to carry out the functions for which they were responsible. He observed 
that the laboratories and centers became aware that the knowledge base on which they 
were to work was weak and that performance skills and technologies were poorly de- 
veloped. He contended that a majority of the labs and centers increased staff appreciably 
within the past two years but that few could yet be said to have capabilities adequate to 
the tasks involved in the accomplishment of their missions. One of the urgent needs 
which Chase identified was to establish systematic programs to increase the capabili- 
ties of those employed. 

In the oft-cited Clark and Hopkins (1969) manpower analysis, it was observed that 
the demand for research, development and diffusion personnel in 1974 will likely be five 
times the 1964 demand, but that the 1969 annual training output is approximately the fame 
as it was in 1964. Though the nature of manpower demanded has also undergone exten- 
sive fluctuation, the field's response to meeting the reed for these competencies has 
been to "replicate in current training programs the relative proportions of personnel 
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found in the 1964 R, D, and D community." The authors conclude by predicting that the 
vacuum in personnel will be filled by whatever talent is available, regardless of their 
qualifications. New products and programs will suffer, as will measurable progress in 
R, D, and D. 

From a recent major USOE publication (Educational Research and Development in 
the United States. 1970) we find a similar pessimism about the numbers of individuals 
who will be able to staff the nation's R and D efforts; 

One inescapable conclusion is that a heavy press currently exists 
on the trained personnel available. Some of this slack has been taken 
up by the entry of personnel into educational research from other 
academic disciplines and from industry. Some has been taken up by the 
addition of a growing number of recent doctoral recipients. A great 
portion has been taken up by on-the-job training of individuals, par- 
ticularly in the field of development, dissemination, and diffusion, who 
have assumed newly identified roles ineducational research and develop- 
ment. Finally, the increase in the manpower utilized is also partially 
explainable in terms of the increased scale of R & D work which has 
contributed to greater cost and a larger number of lower technical roles 
without necessarily creating additional demand for highly trained 
researchers. The manpower supply situation does not appear likely to 
improve very substantially as one looks at the projected outputs of the 
present level of educational research training supported by USOK. 

While the doctoral programs will be supplying 250 to 300 new people 
a year and larger numbers are receiving short-term training, these 
numbers will be insufficient to sustain any large-scale expansion of 
R & D effort. 

Glaser (1966) emphasizes the desperate need for the creation of new roles in the 
field of development; 

In education, the dilemma is that educational researchers are in 
short supply, and the individual trained to do research and the indi- 
vidual capable of doing developmental work is one and the same per- 
son .... At the present time, 1 see little other course than individuals 
doing double duty so that theory, research and practice are mutually 
influenced. 

Evans (1969) notes the inadequacy of current training efforts for the developer whom 
he views as the "aspiring educational technologist" who hasn't enough money to spend 
to receive an M.A. or Fh.D. nor enough time to waste on workshops that are inadequate 
for his ideas. Most of the courses available to him are irrelevant and the possibility 
that a degree-granting institution will offer a diploma in instructional technology is 
slim. 



But while there is a considerable amount of data indicating a major need for training 
programs to prepare product developers and evaluators, the designers of the current 
program were anxious to see whether there would actually be takers, that is, whether 
there would really be those agencies or ind’viduals who would take advantage of our 
consortium-based training program if we were to get it under way. Accordir r ’y, in 
mid -September we sent out exploratory letters to over 300 agencies and individuals 
briefly describing our proposed training program and asking them whether they could 
foresee any likelihood that such a training program would be of use to them. The re- 
sponse was gratifying. Of the 91 responses, 79 agencies or 87 percent indicated a 
definite need for such training programs and a high probability that they would use the 
service. In Appendix B, all names of positively responding individuals or agencies are 
cited. 
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Training Program Rational 



Briefly, the training program is designed to produce educational product developers 
and product evaluators who are results-oriented . A results-orientation will be seen not 
only in the types of skills we provide for trainees, but in the way the training program 
itself is designed and evaluated. Uur concern for process will be guided by outcomes. 
Innovative training vehicles may be interesting, but they will have to yield demonstrably 
superior results with trainees before they will be judged satisfactory. 

The product developers and product evaluators we prepare will possess a similar 
results-orientation. The developers will not be satisfied with instructional materials, 
no matter how attractively packaged or how highly esteemed by scholars, unless those 
materials produce measurable behavior changes in learners. The evaluators will view 
the potency of materials to produce measurable changes in learners as the primary, 
albeit not exclusive, criterion by which to evaluate the quality of those materials. 

The r, results ,r which will be sought by the trainees are, as has been implied above, 
measurable modifications in the behaviors of the intended learners. But lest this con- 
ception of results be considered a restrictive one, it should be noted that the types of 
measurable modifications under scrutiny would be wide ranging and diverse. We are 
interested in far more than whether a learner can make correct marks on an IBM test 
response sheet. The results-orientation we will promote will find our product developers 
and evaluators teasing out subtle changes in learners through a host of both customary 
and esoteric measurement tactics. Some of these subtle learner modifications, for 
example, those dealing with affective outcomes, may be far more critical than more 
classical achievement test dimensions. If we are to prepare tomorrow’s educational 
developers, for example, we must equip these people to consider the affective educa- 
tional goals not yet popular, indeed, perhaps not yet even considered. 

This results-orientation, which we shall be guiacd by as trainers and which we 
shall promote in our trainees, requires that both trainers and trainees become inor- 
dinately skilled in detecting the wide range of results which can reflect suitable learner 
progress. A significant segment of our training program will be devoted to providing 
trainees with such competencies. For the most part, this training can be identical for 
both product developers and product evaluators. But even more importantly, perhaps, 
the consortium training staff must constantly and systematically be searching for new 
techniques of measuring the kinds of competencies, attitudes, interests, etc. which will 
enable our trainees to function more effectively. We must scrutinize our own training 
efforts in light of the most defensible set of results. 

The current national emphasis on educational accountability is quite consonant with 
our aspirations for the trainees as well as for ourselves as trainers. We wish to be 
rigorously judged on the basis of the results the trainees subsequently prodUL We 
want our trainees to apply the same standards. Educational accountability is a pi, rase 
which correctly conveys the orientation of the proposed training program. 

Central to our undertaking as a consortium-operated training program is a com- 
mirment to study the efficacy of this type of training structure. All agencies within the 
consortium possess unique capacities to train either educational developers or product 
evaluators. We believe that pooling the training resources of these many groups will 
produce a markedly more powerful and efficient training program. We believe that the 
basic structure of this training consortium can be applied elsewhere in the U.S. to com- 
parable training problems, But we are results-oriented. We vvBl have to see whether 
our optimistic predictions are realized. In other words, while our chief interest will be 
in carrying out the best possible consortium-operated training program, we shall also 
be carefully studying the worth and potential exportability of the type of training con- 
sortium we have assembled. 




6 



56 



CONSORTIUM AGENCIES 



In this section the various agencies constituting the training consortium will be 
identified and briefly described. In addition, the chief role of each consortium agency in 
the training program will be described. It should be noted that many of these agencies 
are by far the most competent such groups in their respective fields. In the jacket inside 
the rear cover we have included selected brochures, reports, and comparable docu- 
mentation to illustrate more completely the excellence of the consortium agencies. In 
alphabetical order, then, these are the consortium units: 



Americon Tope Duplicotors 

Description . American Tape Duplicators (ATD) is one of the nation’s largest firms 
engaged in producing audio tapes of high fidelity for music and voice reproduction. It 
produces materials for monaural and stereo rcel-to-reel and cassette tapes of various 
sizes and lengths. 

ATD produces, packages and ships all of the audio-tape programmed learning 
materials distributed by Rheem -Califone throughout the United States. It maintains 
sound studios and Is one of the pioneer firms in the production of programmed educational 
software. 

Role . The specific roles of the ATD are as follows: 

a. To train programmers of individual and group materials of programmed in- 
struction for audio tapes, film strips and combined AV hardware to perform the 
functions of: 

1. conceptualizing 

2. scripting 

3. editing 

4 . recording (including the use of studio equipment) 

b. editing of taped masters 

c. mastering of tapes for duplication 

d. packaging and production of packages 

e. shipping and record keeping 

f. field testing and experimental design 



BFA Educotiono I Medio 

Description . BFA (Bailey Film Associates), a division of the Columbia Broadcasting 
System, produv es and distributes educational films, filmstrips, super 8mm loops and 
cartridges, study prints, and multi-media kits. BFA's present distribution is almost 
entirely with public and private schools in the United States and Canada; however, they 
have worldwide representation through both their own sales program and that of CBS 
Enterprises. Bailey Films and Film Associates of California, both well-established in 
the school media business, were acquired by CBS in 1966. Soon after, the two companies 
merged to become BFA Educational Media. BFA now has over 150 employees, Including 
25 sales representatives in the field throughout the U.S. and Canada. 

Of most lelevance to the consortium's training program is the character of the 
Product Development Department and the modus oper-.ndi of BFA with respect to pro- 
duction. Editors, all specialists in curricular areas who possess considerable film evalua- 
tion experience, generate or react to specific film ideas with prospective producers. 
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Kditors then recommend fuiding for productions based on their past experience, feedback 
from school consultants, and information from field sources. Product Development 
personnel are responsible for approval Q f the Initial script and for evaluation and super- 
vision at several intermediate steps. Working in conjunction with BFA's production 
staff, they therefore supervise the overall production. The actual production of media, 
however, is usually done by independent producers. Once a production is completed, 
BFA handles all details of advertising and distribution. BFA has had extensive experience 
in assessing curricular needs of school districts throughout the nation, and in responding 
to thuse needs with media carefully designed for the strengths and limitations on class- 
room use. Members of their Product Development staff have major competencies in 
the production and evaluation of educational media with respect to content, objectives, 
relevancy to the curriculum, intended usage, effectiveness in reaching the intended objec- 
tives, and comprehension of the aesthetic elements of visual communication. 

Role . Briefly, BFA’s major training contributions within the consortium wilJ in- 
volve training in prognosis and in media design. 

Crucial for the product developer is the ability to critique an existing production 
with respect to its probable success in changing viewer behavior. BFA would supply 
training which would enable the trainee to dependably predict the following about a 
production: visual (filmic) impact, clarity of content for the viewer, whether or not it 

will probably affect behavior in the way the objectives state it will, suitability for the 
intended age level, and whether or not it meets a standard of communication for the 
most demanding of teachers. 

Media design, decisions about which format, length, what concepts, etc., follows 
function and need. Much that BFA has to communicate to the trainee about design for 
media will follow mastery of the skills associated with accurate prognosis. In addition, 
BFA's production staff and the independent producers with whom the firm works, will 
help the trainees with specific design problems. 

Trainees will also become familiar with problems and methods of marketing, 
packaging, and distribution which have indirect bearing on final product design. 



CTB/McGraw-H i \ | 

Description . CTB (formerly, California Test Bureau) is now the testing and 
evaluation division of the McGraw-Hill Book Company. It offers a comprehensive range 
of materials and services to schools to meet their evaluation needs for instruction, 
counseling, curriculum management, research resource allocation, and placement. 

CTB publishes standardized, norm referenced tests of achievement, aptitude, 
adjustment attitude, interests and study skills. It also publishes programmed materials 
for sclf-instruction in basic skills. It is planning the publication of a scries of criterion- 
referenced prescriptive inventories in the basic skills and the development of computer- 
based instructional systems. 

CTB provides machine-scoring services for its tests, produ.'ing sophisticated re- 
ports for the individual student, the classroom teacher, the counselor, the school ad- 
ministrator, and the student's parent. 

CTIVs Kvaluation Services include the designing of evaluation programs and t’nc 
providing of evaluation assistance to school districts, 

CTB's Research Services include consultation relative to sampling design, statistical 
analysis of data, and brief descriptions of the results of studies. 

CTB haC more than 20 evaluation consultants, all with higher degrees, vorking with 
school personnel in assigned areas throughout the country. In the home office there arc 
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25 professional employees, also with higher degrees, working on product development and 
evaluation. 

Role. CTB will provide one-semester internships in its home office in Monterey, 
giving the intern an opportunity to participate in all aspects of development of both norm- 
referenced and criterion-referenced tests and related instructional materials, including 
the writing, editing, and analysis of test items, the writing of manuals, the designing of 
norming procedures, the planning of diagnostic and analytic report forms, the design of 
answer sheets, the scoring of tests, the production of test booklets, and the completion 
of a market planning guide. 

The intern would gain experience in the development of instructional materials 
related to the criterion-referenced prescriptive inventories. He would also review 
existing evaluation models, apply the practice of an evaluation model to designing an 
evaluation for a school district, gain some experience with the concept of monitoring 
educational evaluation, item sampling of tests and other specific tools used in evaluation, 
gair, some experience in describing evaluation data and an opportunity to become involved 
in current research studies. 

CTB would also provide staff members for instruction in test development, computer- 
based systems of prescriptive inventories with related instructional material, test 
research, evaluation services, processing of used tests, and production of test booklets. 
Such instruction could be given in a one-week or two-week seminar at the Laboratory or 
at CTB's home office. 



Center for the Study of Evaluation 

Peso ription , The Center for the Study of Evaluation (CSE) is a research and develop- 
ment center funded by the U.S. Office of Education and has been in operation since 1966 
at UCLA. The Center, the only one of its kind focused on the specific analysis of the 
educational evaluation process, has been instrumental in recent years in advancing 
theoretical and practical knowledge regarding the evaluation process. The mission of 
the Center for the Study of Evaluation is to produce new materials, practices, and 
knowledge leading to the development of systems for evaluating education which can be 
adopted and implemented by educational agencies. The scope of activities at the Center 
includes the following: (1) the development of procedures and methodologies needed in 

the practical conduct of evaluation studies of various types; and (2) the development of 
generalizable concepts of evaluation relevant to different levels of education. A more 
elaborate description of the Center’s operation can be located in ?n inclusion (inside 
back cover) which gives the most recent annual report of the Center for the Study of 
({valuation. 

Role. Tlu primary mission of the Center for the Study c’ Evaluation in the training 
program will focus on the preparation of product evaluators, although CSL will also be 
involved in the formative evaluation training for product developers. The Center for the 
Study of Evaluation is frequently called upon to offer assistance to those involved in 
educational evaluation, and while it is not the role of CSE to serve as a consulting 
agency, there have been frequent occasions wherein staff have become con\ersant in 
the practical problems associated with product evaluation. As a consequence of these 
interactions, as well as their more general inquiry into evaluation in its broader aspects, 
CSE personnel will play a prominent role in the training of p r oduct evaluators. 



Consolidated Film Industries 

Description . Consolidated Filin Industries is a major motion picture film proces- 
sing laboratory and a winner of numerous Technical Awards from the Academy of 
Motien Picture Arts and Sciences serving both the entertainment (theatrical) and non- 
theatrieai fielos, While Consolidated Him Industries (CF1) does not produce the films 
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it processes, it does offer consultants w ho will assist a producer who has questions of a 
technical nature, CF1 also has an in-house title and optical department which, using the 
most sophisticated techniques, is capable of innumerable photographic effects and titie 
designs. CFI has been a leader in its field for many years as can best be illustrated by 
the "CFI Is First 11 list which appears in the jacket inside the rear cover. This leader- 
ship and inventiveness ha» attained for CFI an international reputation for excellence 
and the highest possible caliber of workmanship. The producer has a choice of shooting 
his material in either 35mm or 16mm from which, if necessary, 8mm or super 8rnm 
release prints can be made to satisfy distribution requirements. The technical expertise 
that reinforces the feature motion picture producer is made available to the filmstrip 
producer and the non -theatrical film producer. 

Rol e. The role of Consolidated Film Industries can best be described as consisting 
of two main categories: (I) To assist the production company in pre-production, actual 
shooting and the various processing stages of his original film elements; (2) To assist 
in post-production with regard to titles, opticals and art work. CFI f s essential objec- 
tive with regard to the training consortium, would be to acquaint producers in the 
techniques of, and enhance their technical knowledge of, motion picture and filmstrip 
production. 



Educational Development Corporation 

Descriptio n. Educational Development Corporation (EDC) of Palo Alto, California, 
develops instructional materials for all ages of learners. EDC produces programs for 
the school market from kindergarten through college. In addition, EDC develops materials 
for teachers, industry, and the home market. 

Working In close cooperation with consulting experts, the EDC editorial and graphic 
design staffs convert ideas Into products through writing, recording, designing, oil-site 
testing, revising, managing manufacture, and shipping - all according to the needs of 
clients. F.DC's success in developing innovative educational ideas that work, at seen in 
the many products bearing their name, stems from their ability to assess the value of 
an educational idea, attract and manage talented contributors, and organize the resources 
necessary to convert the idea into effective materials of instruction. 

Role. In general, Fducational Development Corporation will supply the required 
expertise of a publisher conversant with the preparation of piinted products. EDC would 
engage directly in the training consortium at least the following three ways: 

A. Assist in devising the training program curriculum by: 

1. Listing criteria useful for trainees in assessing competencies ana needs 
cf learnevs with respect to materials. 

2. Suggesting c riteria useful for trainees in judging appropriateness of audio- 
graphic materials to meet the needs of learners. 

3. Developing a taxonomy of characteristics for different kinds of learning 
materials. 

D. Develop sequences of instruction, including on-eite experience for trainees, 

in the fields of EDC expertise. 

C. Evaluate and suggest modificaticr s for improvement of the t: aining program. 



10 



GO 



Graduate School of Education, University or California, Los Angeles 



Desc rintio n. The Graduate School of Education of the University of California, Los 
Angeles is a major professional school at UCLA and includes primarily faculty members 
in the UCLA Department of Education as well as a limited number of faculty members in 
other departments of the University- The Department of Education within the Graduate 
School of Education is authorized to orfer the following degrees: M.A., M.Ed., KE.D., and 
Ph.D. A faculty of more than eighty professors and a number of ancillary personnel, 
headed by Dean John I. Goodiad, and Associate Dean C. Wayne Gordon, places the School 
of Education in a position to offer not only a number of individurl faculty members who 
are anxious to participate in the proposed consortium, but also the official degree- 
granting capacity of the institution. 

Hole . The Graduate School of Education brings two unique training programs to the 
consortium, both of which have already been established in previous years. The Product 
Research Training Program, established in 1966, offers the masters and doctors degree 
in the field of instructional product development. This program, initially stimulated by 
support from the Educational Research Training Program of ESEA Title IV, Research 
Training Branch, U.S. Office of Education, has already produced a number of doctoral 
degree holders in the field of product development. At the time of its establishment it 
was the only doctoral program in the country specializing in product development and for 
several years thereafter remained alone in this role. 

The graduate program in Educational Evaluation was established in 19 1>8 and offers 
a doctors degree in the field of educational evaluation. This program, closely associated 
with the faculty who work within the Centerfor the Study of Evaluation, prepares doctoral 
degree holders who have special competencies in the field of educational evaluation. 

It is anticipated that both of these programs, because of their degree-granting 
potential, can be coordinated very effectively with the consortium 'based training pro- 
gram. We anticipate that many of the courses now offered in the degree training pro- 
grams can be readily adapted to the proposed training program. 



Institute fo' EducoHonol Development 

Description . The Institute for Educational Development (IHD) received its charter 
as a non-profit educational corporation from the Board of Regents of the state of New 
York in 1965. Its founders represented national leadership in education, industry and 
government. Early assistance in the formulation of IHD as an operating institution came 
from Educational Testing Service and private foundations. In its short history, IED has 
grown largely independent of philanthropic support through performance of contracted 
research and development services on a national basis, to educational institutions, 
government (local, state, and federal), foundations and industry. The principal office is 
i n New York City; the west coast office is in HI Segundo, California, near Los Angeles. 

TED is committed to a search for rational, cooperative and creative changes in 
education, especially in cities and situations affect ng minorities and the poor. The 
Institute was created at a time in history when a critical re-examination of our educa- 
tional institutions was occurring, a process accompanied by a general spirit of dissatis- 
faction in our society at large and in the profession of education itself. Just at that time 
a greatly increased federal engagement in elementary and secondary education was 
taking place. Simultaneously, a new and enlarged Interest on the t?art of industry began 
to he expressed, for reasons of social responsibility as well as self-interest. 

In that context lF r ' was conceived as a new instrument for ’’closing the circle between 
education, industry and government.’ 1 Their work in the past few years has focused in- 
creasingly upon research and development in four general categories*. 

1. Assessing and improving inner-city education. 
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2. Advancing educational technology. 

3. Facilitating relationships between the business community and rhe schools. 

4. Increasing the effectiveness of school organization and administrators. 

Role . Training at 1ED would include involvement in at least one evaluation effort and 
one product development activity. The Institute is currently involved in developing and 
field-testing materials and procedures to supplement Sesame Street viewing in inner 
city homes. 

Through involvement in this effort, trainees would gain competencies in: 

Developing self-instructional home study guides for parents and preschoolers. 
Field-testing and revising these materials. 

Construction and administering criterion instruments designed to assess the effec- 
tiveness of the products and the process. 

Examining the relationship between a mass medium (television) and the individualized 
home study materials. 

Another area of involvement for trainees would include training in project and product 
evaluation as reflected in Special Education and Ri-Lingual education activities. The 
trainees would gain competencies in: 

Design, application, and revision of project and program-wide survey instruments 
designed to assess the products of these training programs. 

Fvaluating disparate product efforts with different goals end objectives. 

Analysis of large data bases containing product outcome da'a for many projects. 



KCET 



Description. KCET, Channel 28, is a public television station operating in the com- 
munity of l.os Angeles with an effective radiated power of 1,200,000 watts, the most 
powerful signal in Los Angeles. The TV signal covers Los Angeles, Ventura, Samp Bar- 
bara, Orange, San Bernardino, Riverside, San Diego, and Kern Counties. KCET is a 
non-profit organization corporately entitled Community Television of Southern California. 
KCET broadcasts pre-school, school arid college level programs during the day, child- 
rens programs in the late afternoon and cultural, public affairs and entertainment 
programs of network and local origination during the evening hours. KCET Is one of 
approximately 200 such stations in the United States. 

Role. Competencies to be obtained by trainees via KCET 's involvement arc dual. 
The KCET staff envisions two basic areas for program development, one in the area of 
developing priorities in instruction and proposal ideas for potential funding, and the other 
in the area of TV or film script'development and production. Participation in production 
would probably be as a segment producer of a larger program format. Anticipated com- 
petencies in program proposal ideas and funding follow. Trainees will be able to: 

L Identify relevant instructional priorities for potential program funding. 

2. Write program proposals appropriate for the instructional priorities de\ eloped. 

3. By reviewing grant proposals that were ■ accepted or i ejected for funding, 
identify segments of the proposals that helped to promote their acceptance or 
rejection. 
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4. Identify effective interest promoting devices that can be utilized in television or 
film production to get and held viewer or learner attention. (Because of the 
vast numbers of people that view TV, most turn on a set to be entertained be- 
cause of their own preconceived ideas about TV. They are usually uncommitted 
to what they are viewing and if bored, can flip to another chanel instantly. The 
product developer has no captive class situation here. This usually implies 
that he has to build attention and transform that attention to involvement which 
can develop commitment.) 

5. Be able to develop appropriate supplemental learning material, if necessary, 
in book, audio, slide, etc. format to promote material review for learner s. 



The Los Angeles County Superintendent of Schools 

Description . The Los Angeles County Superintendent of Schools Cffice is the inter- 
mediate educational unit serving 95 local school districts; the State Department of Edu- 
cation; and numerous county, state, and federal agencies with activities in the educational 
field. The current annual budget is approximately $22,000,000 and the full-time per- 
sonnel number about 1,100. The primary responsibility of the office is to serve in a 
leadership and coordination rote with the districts of the County and to act as a liaison 
between local aistricts and the state. The Office is also engaged in direct operational 
programs in the fields of special education, and special schools for juvenile delinquents. 
Classes am held in 76 locations and served by a staff of 450 tea r hers. 

In addition to the coordination and leadership responsibilities, the County Office 
monitors state and federal programs, provides task forces to assist districts with spe- 
cial problems, initiates and designs research and evaluative programs, and furnishes 
expert consultant help to school district operational procedures. The County Office 
serves under the jurisdiction of a seven member appointed Board of Education, and 
County Superintendent, Dr. Richard M. Clowes. 

Role . The Los Angeles County Superintendent of Schools Office can provide for the 
necessary liaison with school districts in Los Angeles County and, probably, in sur- 
rounding counties. This would provide settings in the real world with which trainees can 
work. In addition, the County Office has many skilled people who could work with the 
trainee and help develop whatever specific skills the trainee is to learn. The Los Angeles 
County Office could also provide meeting rooms, occasionally, where discussions could 
be held. The Los Angeles County Superintendent of Schools his several operating pro- 
grams such as the film library, television, radio, special education and special schools. 
The Ccunty Office also has subject matter specialists. This consortium unit could pro- 
vide information and training in areas of previewing, selection, purchasing, producing, 
evaluating, handling, booking, and the like. 



System Development Corporation 

Description . System Development Corporation (SDC) with headquarters in Santa 
Monica, California is one of the world T s leading developers of information and training 
systems. During its 13-year history, SDC has performed over 1400 contracts totaling 
more than one-half billion dollars. SDC clients include more than 50 departments and 
civilian agencies of the Federal government; military agencies and NASA; several dozen 
state and local government agencies; and numerous universities, hospitals and libraries. 

F or this wide range of customers SDC has designed and developed computer based 
management systems; designed, implemented and conducted education and training pro- 
grams; and, has applied specific skills in such areas as system analysis, training, 
consulting, engineering, simulation and computer programming. SDC's staff of 200 includes 
information processing, human factors and operations research scientists, computer 
programmers and engineers, holding nearly JOOdoctoraL 300 masters and U00 bachelors 
degrees. 
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Work in educational research and development is conducted within the Public Systems 
Division by the Hducation Systems Department (ESD). The KSD staff of 30 professionals 
is currently engaged in projects for the U.S. Office of Education, the Ford Foundation, 
the National Science Foundation, the Job Corps and several state and district educational 
organizations. The ESD staff has accomplished or is presently engaged in work on in- 
structional materials design, computer-assisted instruction, educational data processing 
system design and development, educational program design, development and evalua- 
tion, in-service teacher training, development and operation of educational seminars, 
residency programs cmd short courses. 

Role.. System Development Corporation v;jii teach prospective education developers 
and educators in the use of data processing technology to develop and evaluate educa- 
tional products. At SDC, students will acquire competent in the design, production and 
evaluation of instructional products that either use a computer as a medium- -as in 
computer -assisted instruction <CAl)--or depend heavily on a computer to aid in the 
development or evaluation processes. Trainees in educational product development will 
gain skills in; 

1. production of lesson materials in sciences, mathematics, computer sciences, 
etc. in which the computer is a tool for students to simulate or model a process. 

2. prepai ation and administration of C.Al lessons on computer using author 
languages suchasSDC's own PI ANIT Students will gain experience in analyzing 
content materials, in formulating lessons, putting lessons into a machine and 
testing the materials on actual students. 

3. use of a computer as an nid in generating lesson materials, Programs exist 
or are currently being developed at SDC to generate a large number of specific 
items from a data base of item types and pattern specifications. For example, 

many items of the form: "You get dollars a day for days, How many 

dollars do you get in all? a ~|“T) , c , d or e_ can be 

generated by a machine. A major conTributiorTto be madeT>y~ producfUevelopers 
of the future may be the discovery and specification of item types for machine 
production. 

Trainees in educational product evaluation will gain expertise in: 

1, planning for the use of data processing as an aid in evaluating educational 
materials. Performance data at the specific objective level for many students 
can easily result in a data base with thousands of entries, Data processing is 
a necessity in such cases. 

2. defining categories of information to be abstracted from a data base is an 
important evaluation skill. SDC will make computer time and many of its data 
management systems available so that a student can gain experience in '’massag- 
ing" performance data. 

3* using an existing instructional management system that was developed for the 
Southwest Regional Research and Development Laboratory, This system provides 
a language for describing courses; scores and analyses mastery tests using an 
optical scanner and computer; prescribes remedial instruction; ana provides 
several types of management reporrs that can serve in evaluating instructional 
materials, 



Deportment of Theater Arts, University ot Californio, Los Angeles 

.Desc ription . The Department of Theater Arts embraces theater, film, television, 
and radio and has full facilities for instruction in all of tnese areas, The Theater Divi- 
sion has two thearers and a puppet workshop, all capable of being used for film and 
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television. The Film and Television Division includes three large sound stages, 35 
editing rooms, mixing room, two projection rooms and a projection theater, three 
television studios, two mobile units, and an animation workshop. Because ot its strategic 
location near Hollywood, the Department attracts many students with professional ambi- 
tions, and this is reflected in the high calibre of their work. The Department has won 
several international awards for its student films and is widely recognized for the high 
level of its work in theater and television production. Devoted entirely to instruction in 
the theater arts, the Department has a faculty of 53 full-time teachers, 30 teaching 
assistants, and lists over 1 50 separate courses. 

Role. Generally, the idea is that intensive skills classes and workshops would be 
conducted at the film and television studios of the University. The internship program 
would be provided by the cooperating professional agencies which are not set up for 
training but which offer splendid opportunities for the trainee to observe the workings of 
an ongoing professional operation. All trainees engaged in product development would 
take tha workshops in writing, film, and television production. In addition, they would 
be given courses in evaluation and utilization as well as observation experiences under 
the internship program. 

It is important that early in the year all students in product development have the 
experience of visiting several classrooms to see film, television, and other teaching aids 
being utilized in the classroom. As a consequence, they should be able to conceive of 
their products not as studio productions, but as presentations before a classroom, and 
imagine hypothetical pupil reactions to various elements of their material, When their 
working product is completed, hypotheses can be actually tested and the validity of their 
hunches can be confirmed or rejected. 



Regional Educational Laboratories 

The consortium will wish to use developmentally oriented regional laboratories pri- 
rrarPy as internship settings. Three regional laboratories in relatively close proximity 
to the consortium nave indicated a willingness to establish such internships for our 
trainees. These are the Southwest Regional Laboratory for Educational Research and 
Development in Inglewood, California; the NorthwesL Regional Laboratory, in Portland, 
Oregon; and the Southwestern Cooperative Educational Laboratory in Alburq jerque. 
New Mexico. 

It should be noted that a number of key staff members in the proposed consoriium 
have an excellent working relationship with the nearby Southwest Regional Laboratory, 
principally, Drs. Eva Baker, Evan Keislar, John McNeil, and James Popham, all of whom 
have served either as consultants or as members of that laboratory's professional staff. 



TRAINING SEQUENCES 

Because of the two distinct emphases of the proposed training program, i.e., product 
development and product evaluation, ii will be more clear if the training sequences 
associated with each are detailed separately. We sh*.!i first examine the sequences 
designed for product developers, then those for product evaluators. In a subsequent 
section the specific objectives will be presented along with a description of the measure- 
ment procedures to be used with each objective. 



Training Sequence for Product Developers 

There will be two basic sequences designed to train product developers. Although 
f here are other differences between these two sequences, we can draw a distinction on 
temporal grounds alone, namely, long term and short term. 



o 

ERIC 



65 



The long term sequences will occupy one full-time calendar year for the trainee. 
The bulk of the training will take place either on site at the location of the consortium 
agency's plant/office or at the Consortium Training Laboratory. Short term sequences 
will be ad hoc in nature and will range from a few days to a few weeks in length. Whereas 
the long term training sequence will intensively cover all phases of development, the 
short term effects may focus on only a single phase, e.g., product tryout. 

On Site Training . Certain training for developers must of necessity be undertaken 
where physical facilities permit the provision of realistic experience with various sorts 
cf equipment. For instance, the developer who wishes to become facile in the use of 
educational television must really learn to work with the most current television equip- 
ment. Promotion of desired competencies cannot be accomplished exclusively through 
the use of textbooks. Thus,, consortium units such as Consolidated Films Industries, will 
design and conduct one phase of the training program at the CFI plant in Hollywood. 
Economic considerations will be combined with availability of competent training per- 
sonnel to govern the number and duration of on site troinirg components. We must be 
judicious so as not to interfere with the normal operations of consortium members. Yet, 
there ar^ some sorts of training which definitely must be conducted on site and could not 
be carried out at the Consortium Training Laboratory. Each of these will be subsequently 
described in detail. 

Most training will occur in the Consortium Training Laboratory. Since its operation 
will be integral to the proposed training program's success, a description of how it will 
function and, indeed, why it is necessary will not be provided. 

A number of key staff members in the consortium have considerable experience in 
training educational developers, both within the more common confines of a university 
setring as well as within mission oriented agencies such as regional educations’ labora- 
tories. There are significant disadvantages in training developers in either o f these set- 
tings. To avoid these disadvantages we wish to establish a separate training laboratory 
with its own special goals, namely, the preparation of educational developers and product 
evaluators. 

Training in the Universi ty. What are the problems of training individuals within a 
University setting? The principal problem is that the university has had to devise mechan- 
isms, administrative and otherwise, which are suitable fordealing with the diverse array 
of educational programs carried on within the University. There are constraints, for 
example, dealing with the modification of course content. Changes must be approved by 
the properly designated faculty and administrative monitoring groups. In general, these 
mechanisms are good, for they prevent curriculum abuses. Yet, they are most pertinent 
for established disciplines where content structure has been rather well established. 
For fields where content clarification is emerging, such as is the case with respect to the 
fields of product development and evaluation, these mechanisms prove cumbersome. 
To illustrate, what if we wish our trainees to secure the skills provided in only about one 
fourth of a standard statistics course? Can we enroll them for 25 percent of the course? 
Obviously not. There is a tendency under such circumstances for the faculty adviser to 
Sign a student up for the whole course, rationalizing that the rest of the course (the 
irrelevant 75 percent) will be "good for him." This correctly illustrates the cifficulty 
of tailormaking a training program for a new specialty within the university setting. 
The real or somttiines imagined barriers to particularization prevent one from design- 
ing a truly relevant training sequence for new specialties. 

This should not suggest that there are no particularly useful instructional experiences 
available from the university which are just right for our program. Certain course offer- 
ings, for example, UCLA'e Education 433A, Instructional Product Development, are 
obviously on target. Yet, to conduct the major training on campus would not permit the 
requisite flexibility for the training effort. We need a separate training site such as the 
Consortium Training l aboratory. 



Training at a Regional Laboratory? What about conducting the training primarily at 
the location where product development is taking place, for example, at a regional edu- 
cational laboratory? Surely, since the chief mission of several regional laboratories is 
to produce products, then it would make sense to conduct training for developers at 
the place where development is occurring. But it is unwise to carry out training at a 
mission oriented regional laboratory. What appears to be an advantage is precisely the 
deficiency which precludes the conduct of a truly effective training program. The primary 
mission of the regional laboratory, namely, development, gets in the way of the training 
of developers. Understandably, an institution set up to develop products will focus its 
energy, its best staff, and its major funds, on developing products. Training of developers 
is viewed as a luxury that may divert key resources from the primary mission. 

This Indictment is not based on idle speculation. An actual account of a staff training 
effort at a USOE supported regional educational laboratory will be illustrative. During 
1966 and 1967, shortly after its establishment, one regional laboratory undertook an 
ambitious program to train its development staff. Most of the people hired for the new 
laboratory were bright, but possessed classical educational research backgrounds and 
precious little conversance with the development process. Detailed sets of product 
development objectives were devised which reflected skills to be possessed by the com- 
petent developer, test measures to assess each objective were prepared, sets of new 
self-instructional materials to cover most objectives were written, a glossary of tech- 
nical terms was produced, and series of related readings dealing with product development 
were identified. These materials were coordinated so that a newly hired laboratory 
employee would have to take a pretest covering the objectives, then be given a diagnostic 
profile sheet indicating which self-instruction programs should be read in depth and which 
could be completed in abstract form if his pretest performanc e had been sufficiently 
high. At the conclusion of his study the new employee completed a posi test covering the 
same objectives and once more was giver, a profile sheet showing needed areas of 
remediation. 

The lab's staff training for its developers was working well. People vvera beginning 
to use the same language in describing developmental tasks. Many of the recommended 
cevelopment tactics were being employed. But the training took one whole week! Second 
level laboratory administrators resented the fact that their newly employed staff mem- 
bers were spending a week on concerns other than development. Thus, in 1968 a gradual 
erosion of the staff training program took place whereby staff training responsibility 
was given to the directors of administration sub-units. For a time some of rhece people 
tried to carry on systematic training of their people, using the tests and self-instruction 
materials which had been prepared. As the months wore on, however, the intensity of 
staff training degenerated to the point where it was sporadic at best and nonexistent for 
the most part. 

This actual account of an aborative attempt to train developers at a regional educa- 
tional laboratory was not presented with a view to criticize any laboratory's leadership. 
They succumbed to legitimate pressures for mission accomplishment. In such situations 
training becomes a badly treated stepchild. And this abortive training effort occurred in 
a particularly strong regional laboratory with extremely enlightened leadership. It 
wasn't the laboratory's fault; it was in the nature of the mission orientation. 

It is imperative, therefore, that training occurs where there is plenty of real develop- 
ment actively underway, but where the primary mission is to train dev elop ers a nd pro- 
d uct evaluators . This will be the situation at the Consortium Training Laboratory. 

The Consortium Training L a borat ory. Located off campus but near enough to UCLA 
to draw upon the staff resources of that institution, the Consortium Training Laboratory 
will be the place where the bulk of training takes.place. A number of multiple office- 
facilities are available near UCLA at quite reasonable renral prices (circa 85 cents per 
squaie foot) for one year or longer lease arrangements trie of the 'e locations with easy 
access parking for visiting faculty from consortium member staffs will be selected for 
the Laboratory. 
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There will be several interesting features cf the Laboratory. For cne lluru. t iere 
will be product development activities underway at all times. But the princip T mission 
of the Laboratory will be to instruct trainees, not prepare products. The Laboratory will 
prepare instructional materials in several media to both provide realistic experience 
for trainees and, after two or three years, to produce avenue* through sales so that the 
training program can be partially self supporting. W'c will produce matenaL , dealing 
with the general field of evaluation at the outset, and sell these through a commercial 
puolisher on the basis of whatever USOE royalty policy is currently in effect. Ai the 
present, for example, half of the royalties earned from the sale of such male; ia Is c« uld 
be returned to the training program to defray costs. Yet, In a choice between doing some- 
thing that is good for the products under development or good for the trainees under 
preparation, we will always opt for the latter. 

A second Mature of the Laboratory will be its modular nature. While piodcct de- 
velopment is net, as some have propounded, a linear process, there are ci> r incuLhable 
elements within that process. As far as possible these distinguishable eler. crus will be 
isolated not only with respect to their operation in the development of products, hi f also 
as phases of the training sequence. By having each training and developrm .t cL cent 
essentially separate, we can introduce long-term and short-term trainees t com- 

petency they need at that time, not await the beginning of a complete new training le. 

Another feature of the Laboratory is implied by its name, the Consouiur w i raining 
Laboratory. We have not named it the Consortium Training Center or some comparable 
title which conveys the notion that it is here training will take place. No, we wanted a 
name that accurately suggests training will take place and be studie d. We will be con- 
stantly analyzing the efficacy of the training tactics vve employ. As indicated earlier, 
our resu\ts-orient?.tion will permit the evaluation o f ovir training efforts. We wish to 
evaluate the merits of many approaches to the ^raining of developers and pioduct 
evaluators. Such comparative experimentation will take place almost constantly within 
the Laboratory. 

A final noteworthy feature of the Laboratory is that it will be staffed by an outstand- 
ing group of instructors. The full-time and part-time staff of the Laboratory will consist 
of a uniquely qualified group of educational developers. Not only will there be university 
theorists and researchers, public school personnel with practical educational experience, 
but representatives of the agencies where educational development actually takes place, 
e.g., American Tape Duplicators and Bailey Kiln; Associates, The mix of these diverse 
specialities, all with ihe common goal of producing top quality developers and evaluators, 
will make the laboratory by all odds a most stimulating center of intellectual activity 
regarding development and product evaluation. Results cf staff seminar deliberations, 
training methods experiments, and related technological advances w.Jl be distributed as 
occasional papers of the training program. 

In addition to the staff members representing the various consortium units, most of 
whom will be participating in the operation of the Training Laboratory on a part-time 
basis, we have assembled an outstanding collection of specialists throughout the nation 
who have, in response to a formal invitation, indicated their potential willingness to work 
with us in operating the training program. Most of their input will focus on the operation 
of the Consortium Training Laboratory where, quite likely, some will serve as visiting 
instructors. A complete list of these individuals will be supplied later in this document. 

Practicum . Beyond the numerous practicum experiences which will be provided in 
the Consortium Training Laboratory, trainees will be provided with on-the-job super- 
vised internships, particularly those trainees in the long-term sequences. Reasonably 
long short-term trainees will also be given such opportunities. We already have a fine 
internship arrangement developed with SWRL. Dr. Laurence Fish, Director of the 
Northwest Regional Laboratory has also indicated that he will welcome internship 
assignments in that institution. Dr. Harry Shoemaker of the American Telephone and 
Telegraph Company, Inc. has also promised the availability of internship assignments. 
The c e are only illustrative cf the kinds of practicum assignments which will be required 
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of all long-term trainees. Some will extend for only a month or so. Some will last a 
half-year or longer depending on the trainee's measured competencies at the time and 
upon the potential productivity of the practicum assignment. Individual judgment will 
have to be made regarding each of these assignments, of course. Hopefully, short-term 
training sequences of a month or so will include at least brief practicum assignments 
either within the consortium agencies themselves or in other units, e.g,, regional 
laboratories, identified for that purpose. 

The specific ingredients of the training sequences, both short-term and long-term, 
will be given below under product development objectives. 



Training Sequences for Product Evaluators 

To a considerable extent there will be similarities between the sequences we will 
provide to train product evaluators and product developers. While the similarities will 
be briefly noted, the major differences will be identified in this section. There will he 
long- and short-term training sequences. The training will occur both on site at certain 
consortium agencies, particularly within the Los Angeles County Schools, and at the 
Consortium Training Laboratory. Training components will be modularized as much as 
possible. We will call upon an outstanding staff of visiting instructors to supplement 
our regular training staff. Now, what about the differences? 

Since the role of a product evaluator is clearly different from that of a product de- 
veloper, there will obviously be substantive differences in the competencies which we 
shall promote for our trainees. In general, whereas the evaluative shills of the product 
developer are related to formative evaluation, those of the product evaluator aie related 
to summative evaluation. The product evaluator we have in mind will be an individual 
who can assist educators in reaching decisions regarding already developed educational 
products, irrespective of whether they were developed by a rigorous results-oriented 
technology or by a more intuitive author’s-choice procedure. Thus, his skills will be 
based on his abilities to appraise the other fellow's product, not improve his o.vn. He 
will have to be able to infer objectives for materials which do not possess them. He will 
have to be able to secure learner post-instruction performance data as well as other 
pertinent information, e.g., costs, teacher reactions to material, longevity of products, 
etc., so that decision-makers can reach more prudent judgments regarding the materials 
under consideration. 

A n instructional materia ls evaluation service. In the Consortium Training Laboratory 
one somewhat unique activity would be the provision of a limited scope service for those 
who wish assistance in the evaluation of instructional products. We will publicize the 
existence of this service by the training program, and will charge reasonable fees for 
the service to partially defray training costs. T 'er, we will be extremely selective in 
choosing such projects. This is not intended to be a major consulting service, but rather, 
to be analogous to the product development activity w^ shall engage in primarily to train 
developers. Hence, we shall be conducting evaluations primarily to train product evalua- 
tors in a real life setting. 

As we shall see, certain components of both training programs will be taken by 
trainees in each group. For example, training sequences dealing w*th the development of 
mreasures will be applicable for both groups whether the objectives were formulated by 
the developer trainees or inferred by the evaluation trainees. 



The One Veor Program 

The year-long program in product development will be composed of changing in- 
structional formats to suit various emphases during training. Of central importance is 
the access trainees will have to consortium staff. Provision will be made for weekly 
sessions to allow trainees and staff to express perceived needs for modifications. While 
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substantively different, the general structure of the year-long program for evaluators 
would be comparable. 

During the opening weeks of training, each participant will undergo orientation Co 
the procedures and attitudes deemed appropriate to development. The result of a pretest 
on in f ormaticn -based objectives may permit the trainee to investigate more thoroughly 
various rationales for training, based upon guided selections from an extensive bibli- 
ography. The second phase of training will emphasize preliminary application of product 
development procedures. Small teams of trainees, led by a staff member, will engage in 
practice in various development activities, e.g., specification, using simulated conditions. 
Semi-weekly seminars will permit the reasoned interpretatin of difficulties which 
trainees experience. A third phase of training would allow trainees to opt for various 
specializations on an internship basis, e.g., film making, television production, computer 
managed in, .ruction. The outcomes of such specialized internships are intended to provide 
the novice with only minimal competence. If extended training were desired, arrange- 
ments may be made on an individual basis. During Phase Three, trainees would continue 
their relationship with the Consortium Training Laboratory staff by means of independent 
development projects and seminars. 

The fourth phase of training entails a heavy internship commitment by the trainee. 
Internships will be arranged in participating agencies where adequate supervision is 
available. In some cases, the Training Laboratory scaif would have to contribute to the 
supervisory capability of the institution in which the internship is located. Internships 
would involve contractually agreed upon tasks for which the trainee would be accountable. 

A fifth phase of the program would focus on development of a criterion product by 
which the trainees skills would be appraised. Trainees would select from alternative 
topics and formulate, develop, test, revise, etc., a discrete instructional product. The 
short-term effects of the training will be evaluated through examination of the completed 
product and accompanying technical report. 

Presented schematically, the year-long training program can be depicted as follows: 



One Year Program in Product Development 







EMPHASIS 


DOMINANT FORMAT 


DURATION 


Phase 


I 


Orientation 

information 


Lecture -discussion; 
independent study 


1 


month 


Phase 


II 


Controlled 

practice 


Team-based and individual 
simulation; seminars 


3 


months 


Phase 


III 


Specialized 
T raining 


At least 75% activity in produc- 
tion in areas of interest, e.g., 
television, computers, individual 
development, seminars 


1 


- 3 months 


Phase IV 


Internship 


Integration into development 
agency; responsibility for 
specific tasks 


4 


- 6 months 


Phase 


V 


Evaluation 


Independent development 
of a criterion project 


4 


- 6 weeks 



Short-Term Troining Programs 

A variety of short-term programs, from one week to one month in length are en- 
visioned as appropriate for the training program. Programs may be specific to particular 
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activities associated with development or evaluation programs, e.g. f fie testing, may 
relate t o media, or may be a telescoped version of the orientation inforn.atioii portion of 
training. They may also be directed to homogeneous trainee groups, e.g., state Title III 
evaluators or editors in a publishing house, "’articular agencies, for instance, American 
Telephone and Telegraph, have displayed interest in such training for a major portion of 
their development staffs. 

Short-term programs, whether specialized in terms of development procedures or 
niei’a, or comprehensive and compressed, will be formulated as needs arise. The base 
for such programs will generally be artful combinations of extant competences. 



Graduate Degree^Re loted Pragrams 

Doctoral T r aining . A doctoral degree may be secured is part of the program pro- 
vided certain criteria 9 ' e met: 

L The trainee is admitted to the UCLA Graduate School of Education. Admission 
for graduate work requires a master's degree, a Graduate Record Examination 
score of approximately 1,2CQ, a grade point average of 3.0 from a Class A 
institution. 

2. Ihe trainee completes additional requirements as stipulated in the programs ot 
Instructional Product Research or Educational Evaluation, including course - 
work, qualifying examinations, and dissertation. 

It is anticipated that such requirements would require approximately eighteen months 
to complete. Sample, not exhaustive, competencies required for the completion of the 
doctorale in Instructional Product Research in addition to those specifically required for 
the consortium training program are as follows.- 

Research 

1. Ability to design experiments and evaluations, including: 

a. adequate conceptualization of manipulated and demographic variables and/or 
adequate description of program to be evaluated. 

b. ad^ jate production and validation of criterion instruments 

c. appropriate selection and application of statistical procedure. 

d. qualified interpretation of results. 

e. literate reporting of results. 

Prerequisites would involve ability to analyze extant research in terms of dimen- 
sions a - e. 

2. Ability to identify and :oimulate alternative research approaches to problems 
in development and instruction. 

D evelopment 

1. Ability to identify aiiernate procedures for each major development activity. 

2. Ability to analyze the role of development in education with respect to basic 
research in the disciplines, banlc educational research, applied research, 
dissemination and practice. 
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3. Ability to relate educational development problems to those faced in other 
product development fields, e.g., engineering* pnarmaceuticals. 

Such competencies are evaluated on written qualifying examinations. Course pat- 
tern expected in product development is as follows: 

Core Courses 

210A Analysis of Educational Research 

2K)B Experimental Design in Educational Research 

258A Problems in Instruction il Research 

260 Seminar: Principles of Curriculum and Instruction 

267 Seminar: Educational Technology 

418A Fundamentals of Programmed instruction (or) 

419A Experimentation on Media of Communication a: 1 Instruction 

423A Principles of Curriculum and Instruction 

420B Instructional Analysis 

420C Evaluation of Curriculum and Instruction 

433A Instructional Product Development 

Recommended electives (three courses required) 

210C Experimental Design: Advanced Topics 

211 A The Measurement of Educational Achievement and Aptitude 

21 IB Measurement in Education: Underlying Theory 

212A Learning arid Education 

212B The Teaching of Concepts 

212C The Teaching of Problem-^ living Abilities 

(Within the Ed.D. a stude might choose to emphasize product research 
and evaluation techniques or production of materials. The necessity for a 
minor would depend upon the candidate’s background,) 



Masters of Ar's Degree 

Trainees may pursue masters degrees provided they are admitted to the UCLA 
Graduate School of Education as above. They must complete nine courses and a thesis, 
but do not receive degrees strictly In product development or evaluation. Since the M.A. 
program is very flexible it would be possible to complete work for the masters degree 
with approximately one quarter of additional work. Masters 'hoses may be results of 
development effort. 



C;nerol Corr pete n- ies (Or the Educctionol Evaluation Doctoral Program are as follows: 

1. The stuaent understands the use of evaluation *md the n ature and rationale of 
diff e rent evaluation strategies . This includes identifying situations wTTch require e TaTua- 
tion and the characteristics, attributes, similarities, limitations, advantages, and impli- 
cations of alternative evaluation models (or theories) fo? different kinds of evaluation 
problems. 

2. The student understands the nature and role of objectives . This includes identi- 
fying objectives (or goals), explicating the consequences of them, translating them into 
behavioral terms when necessary, comparing actual and intended outcomes, identifying 
dir '.repancies between goals and outcomes, and recognizing the impact of unanticipated 
outcomes. 

3. The student can Identify and explain the major strengths, weaknesses, and im- 
plications of various kinds of evaluation measures and other information collection tech- 
niques. This includes understanding of and/or knowledge about issues such as validity, 
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reUahility, test construction philosophies and strategies (e.g., norm vs. criterion refer- 
ence '), survey research techniques, score reporting and interpretation proced ures, kinds 
of measures and data needed for different kinds of evaluation problems, and sources of 
information about measures, data, and techniques. 

4. The student can identify and explain the strengths, weaknesses, and implications 
of different kinds o f evaluation designs and analyticaltechniques. This includes determin- 
ing the appropriateness of designs and methods for obtaining needed summary data for 
different kinds of evaluation problems, such as evaluations of educational systems, instruc- 
tional programs, students, and non-pupil progiat.iS (e.g., TT D. This also includes 
understanding statistical and other inference technique.' (r g., computer simulation). 

5. The student understands how to communicate wich decision-makers and others in 
order to implement evaluation designs and translate results obtained so that appropriate 
actions can be taken. This communication requires understanding the interrelationships 
among evaluator, decision-makers, and others involved in and affected by the evaluation. 

6. The student can apply all the foregoing knowledge to actual and hypothetical 
evaluation problems. This involves describing and carrying out all rhe step-by-step pro- 
cedures and techniques we would use, includingdata collection, analysis, and communica- 
tive procedures. Competencies are evaluated on written qualifying examination and in the 
dissertation. Anticipated course requirement will be selected from the following: 

210 A Analyses of Educational Research 

210 13 1 xperim^ntal Design in Educational Research 

210 C Experimental Design, Advanced Types 

211 A Measurement of Educational Achievement and Aptitude 
21113 Measurement in Education: Underlying Theory 

420 C Curriculum Evaluation 

255 A Seminar in Evaluation Theory 

255 B Seminar in Evaluation and Decision-Making 

259 Evaluation in Higher Education 

R ccom mendcd Courses 

J 33 A Instructional Product Development 

420 A Curriculum 

200 A Survey Research Methods 

203 Philosophical Interpretations of the Behavio; il Sciences 



INSTRUCTIONAL OBJECTIVES AND MEASUREMENT PROCEDURES 

We turn no* to the specific goals of the training program. Since this program, if it 
is funded, will be producing crucial educational practioners, for their influence on the 
future may indeed be profound, we wint to be inordinately careful about the kinds of 
trainees we are producing. We will esclew preparation practices which will pioduce in- 
dividuals trained only for today and not tomorrow. Instead, we will endeavor ti produce 
product developers and product evaluafors who can func ; on with great skill today, but 
who will be ready to play a role in future development and evaluation configurations not 
yet anticlpatable. In this vegard, we will be preparing trainees who possess (1) an 
empirical orientation, (2) adaptability skills, and (3) specifiable competencies. The first 
two of these three dimensions will be nurtured in a general fashion throughout the training 
program. We will attempt to become skilled in assessing the attainment of these two 
general goals, but will clearly be in a better position to assess specific competencies. 

All three general gaals, however, will guide our training efforts. 



An Empirrcal Oricnto l! on 

Gagne (1969) writes of the "irreducible" characteristics desired of an educational 
technologist. These characteristics fall in three categories: values, knowledge, and 

methodologies. Gagne emphasizes that a central value for educational developers to hold 
is the belief "in empirical evidence as a source of truth and a preferred basis for action." 
Glaser (1966) has noted that such empirically-oriented development personnel likely hive 
been trained as educational researchers and often act in both research and development 
capacities. 
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Adoptibilh/ os a Desired Attribute 



Kuhn (1966) describes manpower lessons inferred from the study of management 
problems in the emerging nuclear power industry. His observations have great relevance 
for instructural product development and product evaluation iii their current pre-tech- 
nology state. 'Kuhn proposes a framework within which one can assess manpower in 
developing technologies. His assumption is that requirements in the early stages of a 
technology change in dramatic and in largely unpredictible ways. The flexibility oi the 
Jeveloper along ary one of three dimensions is critical to his ultimate usefulness in a 
rapidly accelerating field. Individuals with flexibility along Kuhn's hypothesized first 
axis demonstrate their ability to draw from diverse theories or experience to design 
solutions to given problems. Such persons would correspond in role to those responsible 
for prototype preparation in the product development cycle described. Individuals with 
flexibility along a second axis might instead be able to fulfill a given function in a wide 
variety of problem areas. For instance, he would be able to prepare specifications in the 
humanities as well as in the sciences, for pre-school learners or for college students. 
Those who exhibit: flexibility along a third dimension are most prized by Kuhn: "This 
axis is parallel to tlv flow of technological development; a man flexible in this direction 
is able to shift the ■ nter of his job concern as the emphasis of a technology progresses 
from concept to rnai stable product." These individuals are rare and are essential when 
the functional distinctions among development stages are not discrete. Another factor is 
that manpower requirements within institutions change. In the early stages of a technology, 
most manpower will be concentrated on developing alternative designs. Later, as the 
technology develops, individuals will be needed for prototyping, field testing, and imple- 
mentation. Rather than idle or dismiss staff as the development focus shifts, agencies 
can use the developer with "parallel flexibility" since he can alter his role as necessary. 

If one accepts Kuhn's analysis, the next puzzle is how one identifies or trains indi- 
viduals with such flexible predilections. One might venture that the training of concept 
synthesizers, Kuhn’s first type, may be most difficult. Training might, however, be 
reasonably provided to promote flexibility in the second and third dimension. Trainees 
could be given experiences at either one or two stages and thus develop a functional 
role, such as test writer or revision maker. In training people to demonstrate "parallel 
flexibility," opportunities would be given to supervise a variety of projects from initial 
phases to completion. 

The type of training needed is, of course, relevant to the organization of the develop- 
ment institution. Organizations structured by functions would have greater need for 
developers with oractice i;i the second axis. Project centered organizations might better 
use developers prepared to exhibit parallel flexibility. 



Specific Competenc i.s nrd Measurement Procedures 

For ease of presentation we have coordinated the presentation of the instructional 
objectives with the measuring devices used to assess the attainment of these objectives. 
Further, these objectives and measurement procedures will be presented separately 
for the two major training program emphases, that is, product development and product 
evaluation. 

It is particularly Important to note that the objectives to be presented in the re- 
mainder of this section represent terminal objectives rather than eg route objectives 
and it should not be inferred that en route objectives (and measures of their attainment) 
will not be important in the training program. For instance, many of the objectives to 
be presented below describe the trainee's ability to generate rertain kinds of material, 
specifications, etc. For instance, the trainee may be called upon to produce a set of 
instructional specifications which satisfy clearly explicated criteria. Now it is highly 
likely that the trainee would first be called upon to identify deficiencies in sample in- 
structional specifications. That is, he would be presented .*ith real or fictitious sets of 
instructional specifications and asked to identify the strengths and weaknesses within 
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such documents. This kind of discrimination practice will, in general, precede most of 
the objectives to be found in the following section. The point we wish to emphasize is 
that for parsimony we have not included a description of all of these en route objectives. 
But they would certainly be included in the raining program, for measuring trainees b\ 
attainment of such objectives we could determine where modifications in our training 
sequences should be undertaken. It should a _o be noted that the highest priority objectives 
have been identified with an asterisk. The reader may wish to attend more carefully to 
such objectives. 



Product Development Objectives 

At the close of identifiable segments of the training program, the product develop- 
ment trainee will be able to accomplish the following objectives: 



For mulati on 

*1. Ob jective . To design and/or carry out a step-by-step procedure to oe followed in 
conducting an educational needs assessment whereby instructional goals are selected 
on the basis of systematic consultation with reference groups representing the 
community, the learner, and the subject discipline. 

Measurement Procedure . 

A. Objectives will be assessed by 

1. Requiring trainees to write out a procedural description of hew an 
educational ne-^ds assessment should be conducted, given real or fic- 
titious data regarding the situation in whuch the needs assessment js 
to be conducted. 

2. Having trainees engage in actual conduct of an educational needs assess- 
ment. 

B. Criterion 

1. Congruence of written description or actual operation with one or more 
recommended systems for implementing a goals-determlnation analysis, 
e.g., after Popham (1969) or recommendations of market survey con- 
sultants to be employed by the consortium. 

*2. Objec tive. To design and/or carry out a market analysis in which competitive in- 
structional products are identified and tneir external characteristics (e.g., cost, 
time, requirements, needed entry behaviors) contrasted with those of an intended 
product to be developed. 

N lea sure m e n t P rocedure . 

A. Objective ^jH be assessed by 

1. Trainee's ability to describe procedures to be followed in conducting 
such market analyses. 

2. Trainee's identification of potentially competitive products and their 
characteristics given directions to locate all members of a class of 
competitors, for example, all self-instruction products purporting to 
provide basic skills in geography for elementary school children 
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B. Criteria 

1. Procedures mvst include consultation of all standard sources through 
which extar.t products can be identified, e.g., Henderschott’s Programmed 
Instruction bibliographies. 

2. Ac tual identification of nearly all, i.e., circa 90 percent, major com- 
petitors. The competitive products will have been previously identified 
by exhaustive research on the part of consortium staff. 

3. Objective. Given .several desired instructional objectives and comprehensive infor- 
ma tiorTTegar ding extant instructional products, the trair ee will be able to generate 
a defensible rationale for selecting a certain objective (or objectives) for which a 
new instructional product(s) should be developed. 

Measurement Procedur e . 

A. Objective will be assessed by supplying trainee with real or fictitious in- 
formation and requiring the preparation of a written rationale statement 
describing the selection of certain objectives. 

B. Criterion 

1. A jury technique will be employed to determine the cogency of the 
reasons offered in the rationale statement. 



Spec if i cation 

A, Obiective. To write operational statements of goals for instructional products. 
These prod icts may range from brief, highly structured sequences to year-long, 
highly complex instructional programs. 

Measurement Procedure. 



A. Objective will be assessed by 

1. Providing general objectives for trainees to convert to operational 
statements. 

2. Inspecting objectives produced in criterion development project. 

B. Criteria 

1. Overt behavior or observable product of behavior must be described, 

2. Statement of content to which behavior applied must be included. 

*5. Obiective. To specify criteria for judging the adequacy of the learner's responses 
to the product's objectives either by describing necessary attributes which a con- 
structed response must display or by estimating in a well- justified statement the 
required quantitative criterion level, e.g., percent of problems correct. 

M easurement Procedure . 

A. Objective will he assessed by 

1 Providing operational goals with a variety of topics for which the 
trainee must add criteria. 
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2. Inspecting objectives produced by trainees in their criterion develop- 
ment project. 

B. Criteria 



1. Criteiia must be stated for cognitive or psychomotor objective, 

2. Qualitative criteria must provide either rules o r examples by which to 
evaluate the adequacy of performance. 



3, Qualitative criteria must project numerical levels which the trainee 
justifies in terms of evidence that supports the levels posted. Examples 
of such evidence might be entry levels required for subsequent learning, 
experts’ estimates of performance levels. 

4. Either correct answers or rules for evaluating a range of responses 
should be provided. 

6. Objective . To write operational statements of goals for instructional products 
calling for higher cognitive processes (after Bloom, 1956) or generalization, (after 
Mechner, 1966). 



Measurement Procedure. 



A, The objective will be assessed by 

1. Requiring in test-like situations, trainees to produce objectives calling 
for higher cognitive or generalization behaviors. 

2. Inspecting objectives produced for a criterion development task for 
higher cognitive or generalization behaviors. 

B. Criteria 



1. Objectives must describe responses which can be reliably categorized 
by instructional experts as either comprehension, application, analysis, 
synthesis or evaluation or as instances of generalization. 

2. Objectives must i dude some qualification regarding the nature of 
the planned instructional sequence to assure that seemingly taxing 
responses will not be memorized during instruction. 

3. Objectives must describe overt responsec or observable products 
produced by students. 



7. Objective . To prepare operational statements of goals for instructional products 
involving affective responses, either subject-matter approaching tendencies, e.g., 
preference for subject matter; socialization, e.g., refraining from interrupting peers; 
or personal development, e.g., self concept increase measured by appropriate 
devices. 



Measuremen t P rocedure . 

A. Achievement of the objective will be assessed by 

1. Providing general topics, e.g,, attitude toward school, which the trainee 
must convert into affective goal statements. 

2. By inspecting objectives generated by trainees in a criterion develop- 
ment project. 
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B. Criteria 



1. Objectives must be operational. 

2. Goals describe.- must be in terms of volitional rather than assigned 
actions of learners. 

3. On self-report devices, anonymity will be required. 

4. Objectives may describe feelings, opinions, value preferences of 
learners. 

5. U is anticipated that the majority of objectives will be classifiable at 
level two {responding/ or level three (valuing) of the taxonomy of edu- 
cational objectives. Handbook II (Krathwhohl, ec al., 1964). 

6. Feasibility of achievement of objective will be used as an additional 
cciteriom that is, the likelihood that given objectives may be achieved, 
e.g., major personality modifications are unlikely, etc. 

*8. Obje c tive . To prepare operational statements of goals for instructional products 
which adequately describe stimulus limits of objectives, so that a domain of content 
is desc ribed from which particular instances for teaching and testing * iay be sampled. 
(See Kriewall, 1969; Nitko, 1970; Hively, et al., 1 968; Sension, 1970). 

Measurement Proced urc . 

A. Achievement of the objective will be assessed by 

1. Providing test-like situations where trainees must produce domain- 
referenced objectives in their subject field of competence. 

2. Inspecting objectives produced for their criterion development project. 

B, Criterion 

1. Objectives will be satisfactory which stipulate stimulus limits for 
responses. Such limits rnay be provided by (a) a rule which describes 
content domain, e.g., all two digit addition rpoblems, any lyric poem 
written by Keats oi Shelley; (b) by listing every element of the set of 
content, e.g., to write analyses of To Che Lighthouse, Porcrait of JL 
Lady, Sister Carrie, My An tonia. 

9, Objective , To make preliminary estimates regarding group performance levels for 
a proposed product, e.g., 90 percent of the subjects, based upon a study of the achieve- 
ment levels of current practice, if any. 

Measurement Procedure. 

A, The achievement of the objective will be assessed by 

1. Requiring trainees to amend statements o* operational goals by adding 
group performance levels and a justification for their estimates. 

2. Examining objectives produced for a criterion development project. 

B. Criteria 

1. Performance levels must either exceed documented achievement pro- 
duced by current practice, substantially reduce cost, or achieve 
additional objectives concomitantly. 
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2. Performance levels must clearly describe the proportion of the group 
for which success is anticipated. 

3. If success in product goals is planned for less than 100% of the 
students, a justification describing the P pes of students who are not 
expected to succeed is required. 



Behavior Anol ys i s 

•10. Obje c tive . Given operational statements of product goals, to generate sub-tasks 
hypovhes'zed to be essential *o the achievement of the teiminc* task. 

Meas u rement Procedure . 

A. Achievement assessed by 

J. Providing in a test-like context sample operational objectives and 
requiring trainees to write statement of posited subordinated tasks. 

2. Inspecting the analysis of objectives used in criterion development 
projects. 

B. Criteria 

1. Subtasku must be stated in observable behaviors or products. 

2. Tasks must bear logical relationship (as perceived by expert judges) 
to the terminal task. 

3. ^asks for which no subordinate structure is produced must be ac- 
companied by an explanation and a statement of what coordinate tasks 
<. re posited. 

11. Objective . To demonstrate ability to classify sub-tasks according to procedures 
developed by Gagne, including tasks which call for stimulus-response learning 

Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. When given in a rest-like situation a set of tasks, the trainee will mark 
each of the task'* as one of Gagne’s classification of learning (1970), 
e.g., verbal chaining. 

2. When asked to generate an objective in a test-like situation, the trainer 
can properly test subordinate skills and label them according to Gagne’s 
scheme. 

B, Criterion 

1- At least two judges will agree on the classification produced by the 
learner. 

12. Objective . To demonstrate ability to classify sub-tasks according to procedures 
described by Mechner, including tasks which call for association, discrimination, 
generalization. 



o 

ERIC 



29 



79 



Measurement Procedure. 



A. Achievement of the objective will be assessed by 

1. When given in a test-like situation a set of tasks, the trainee will mark 
each of the tasks as one of Mechner's classification of learning, e.g., 
verbal chaining. 

2. When asked to generate an objective in a test-like situation, the trainer 
can properly test subordinate skills and label them according to 
Mechner's scheme. 

B. Criterion 

1. At least two judges will agree on the classification produced by the 
learner. 

13. Objective. To classify sub-tasks as either entry behaviors, skills possessed prior 
to instruction, or en route objectives, objectives for which the product assumes in- 
structional responsibiMty, (see Baker, Gerlach, and Sullivan, 1968). 

Measurement Procedure. 



A. Achievement of the objective will be assessed by; 

1 Requiring trainees to label each of identified subordinate tasks as 
entry behaviors or en route objectives. 

B. Criterion 

1. A jury technique will be used to gauge the likelihood that the trainee's 
discriminations represent reasonable assignments as entry or en route 
behaviors. 

*14. Objective . To present a statement of probable best sequence for sub-tasks, justifying 
the task order on the basis of behavioral analysis according to Gagn^, Bloom, 
Mechner, etc., content structure, or some other reliable basis of analysis. 

Measurement Procedur e . 

A. Achievement of the objective will be assessed by 

1. Presenting trainees with a set of subtasks which they are directed to 
sequence. They must provide a written justification for any sequence 
they propose. 

2. Inspecting sequence projected for subtasks in a criterion development 
project. 

B. Criterion 

1. Justification must depend upon at least one of the following: use of 
behavioral analyses structure suggested by Gagn£, M *chner, or Bloom, 
statements by subject matter, experts on content structure, experi- 
mental data relating to possible sequence alternatives. 

15. Obj ectlv - . To produce a pre -instruction test which assesses behavior stipulated in 
terminal objectives, en route objectives and entry behaviors. 
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M ea surement Procedure. 



A. Achievement of the objective will be assessed by 

1. In a test-like situation, asking trainee to produce a test which mea- 
sures tentative terminal, entry and en route tasks. 

2. Inspecting the criterion project for evidence of a pre-instruction list. 

B. Criteria 

1. Test items must be judged by independent analysts to measure stated 
tasks. 

2. Explicit criteria for judging responses should be included. 

3. Test directions should be provided. 

4. Language should be appropriate to learner. 

5. Test length must be reasonable for age level of learner, Procedures 
for item/person matrix sampling should be used where rest fatigue 
is a factor. 

6. If matrix sampling is used, alternate test forms must assess related 
tasks, i.e., tasks hypothesized to form a learning structure. 



* 16, Objective. Given sets of hypothetical or actual data from pre-instruction testing, 
to identify objectives which should be discarded, objectives which need revision, 
and objectives to be included in the product. 



Measu r ement Procedure . 

A, Achievement of the objective will be assessed by 

1. Identifying, when given hypothetical data in a test-like situation, whi'h 
objective should be revised, included, or discarded. 

2. Making revisions of his objectives in his own criterion project based 
upon pretest data. 

b. Criteria 

1. Decision to discard objective will be based on high achievement levels 
of uninstructed groups. 

2 . Decision to reduce criterion levels, either qualitative or quanritive 
will be justified by a written discussion of serious and unanticipated 
deficiencies in learner performance on entry slclls. 

3. Decision to modify objectives must be explicated on the basis of (a) 
achievement of en route objectives (b) new information or concep- 
tualization of the tasks based upon pretexting. 



17. Objective . Given sets of hypothetical or actual data from pre -instruction testing, 
"to determine if the pattern of response to posited terminal, en route and entry 
skills confirms assumptions of dependence, that is, learning structure, (see Gagne 
and Paradise, 1961). 
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Measurement Procedure. 

A. Achievement of the objective will be assessed by 

1. Trainee^ will be presented with data arrayed by success and failures 
for each of a set of tasks. Trainees will be asked to identify which 
tasks, if any, appear to be dependent upon other tasks. Trainees will 
formulate their conclusions into statements of learning structures. 

2. Given the task to verify hypothesized structure, i.e., dependence 
among objectives, student will prepare from data be has gathered for 
his criterion project, a chart in which learners successes and failure 
by task are displayed. He will proceed as in the previous point. 

B. Criteria 

1. Judged likelihood that the trainee f s selection of a learning structure 
is indeed task analytically defensible according to a priori judgments. 

2. Empirical tryout of trainee's structures. 



Prototype Development 

*18. Objective. Given a description of constraints (Geis, 1970, Shoemaker, 1970), st'ch 
as probable context of use, costs of competing products, computer availability, 
intended life span of product under development, and performance specification to 
produce at least two alternate plans for prototype of products whicli adhere to the 
given limits. 

Me asurement Procedur e. 

A. Objective will be assessed by 

1. Requiring the trainee to produce alternative plans for product proto- 
types in a simulated exercise, 

2. Inspecting the alternate prototypes considered in the trainee's terminal 
development task, e.g., computer assisted instruction or computer 
managed instruction. 

B. Criterion 

1. Judged consonance of alternative prototypes with given constraints, etc. 

*19. Objective . Given a set of performance specifications, to develop a detailed instruc- 
tional plan which includes reliance on some of the following instructional principles: 
direct practice, prompting, knowledge of results, use of advance organizers, m; 
magenic procedures, etc. 

Measurement Procedure . 

A. Achievement will be assessed by 

1. Asking the trainee to illustrate the use of specifically given ins: 
tional principles given at least three different sets of spec ificatv . 
in a test-like situation. 

2. Asking the trainee, when given at least two alternate sets of specifica- 
tions, to select one and develop an instructional sequence er- pitying 
those instructional principles he deems suitable. 
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3. Inspecting the instructional plans the trainee provides in his criterion 
development situation. 

8. Criteria 

1. Principles shouM he illustrated in the instruction, not named. 

2. Description of activity should be in sufficient detail to permit knowl- 
edgeable readers to identify principles. 

3. If principles are used which may be freely interpreted by the developer, 
e.g., reinforcement, contiguity, social motivation, the developer must 
present a justification of the particular stimulis selected, e.g., to 
explain the evidence or basis which guided the form of reinforcement 
employed in instruction. 

20. Objective . Given segments of products to state which of a given set of instructional 
principles are included. 

Measurement Procedure . 

A. Objectives will be assessed by 

1. Priving the trainee with descriptions or sample segments of instruc- 
tional programs, and a list of principles and asking them to indicate 
which, if any, are being employed. 

2. Providing the trainee with sample segments of instructional programs, 
and asking them to name principles employed. 

D. Criteria 

1. All instructional principles included in the segments must be named. 

2. For assessment of A 1 above, principles listed but not actually included 
in the program segments should not be selected. 

*21. Objective . To be able to write a justification for the selcrilon for certain media on 
the basis of Instructional principles, e.g., opportunity .o practice, and logical rela- 
tionship of task to media capacity, e.g., necessity for color or motion, (see Lumsdaine, 
1963). 

Measurement Procedure . 

A. Objective will be assessed by 

1. Asking the trainee when given a statement of intended product goals 
and alternate selections of media or combinations of media, to iden- 
tify the media characteristics logically related to the stated goals. 

2. Inspecting decisions made in the prospectus or the criterion develop- 
ment project. 

D. Criteria 

1. Stringent justifications aie required for media selection, estimating 
relationship of cost to anticipated benefit, e.g., censumnability, life- 
span of product, particular capacities of media which has instructional 
applicability. 
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2. The highly selective use of experimental data comparing media effects 
may be included in the justification, if qualifications regarding lack of 
gen^ralizability of results are stated and particular attributes of thr* 
topics, programs and subjects used in the cited evidence are identified. 

*‘■'2. Objective . To prepare complete instructional products, as well as segments of such 
products, which include selected instructional principles. 

Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. Inspecting segments of and completed products trainees produce under 
simulation conditions. 

2. Inspecting product for criterion development situation. 

B. Criterion 

1. All instructional principles included in the segments must be present 
to the satisfaction of staff judges. 

23. Objective . To describe procedures for the majority of the planned instruction which 
are reproducible; if teacher-mediated products are proposed, to provide a written 
plan for determining the reproducibility of the instructional events and the variability 
tolerances between classrooms anticipated. 

Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. Inspecting segments of products designed in simulated situations. 

2. Inspecting actual product generated in criterion development project. 

B. Criterion 

1. Reproducibility Is defined as anticipated lepllcable instructional events 
(see Lumsdaine, 1964). 

*24. Objective . To design short-term experiments to provide usable data to aid in 
selecting among instructional alternatives, e.g., response mode. 

Measurement Procedure. 

A. Objectives will be assessed by 

1. Providing the learner with given instructional problems and a choice 
among variables to manipulate, and requiring her to design an experi- 
ment for the purpose of gaining information. 

2, Providing the trainee with instructional problems where he has to 
identify the variables mou useful to consider and to design an experi- 
ment to aid instructional decision-making. 

B. Criteria 

1 Variable selected or identified in either A 1-2 above will be mani 
pulable. 
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2. Levels of the variable will be explicitly defined. 

3. If media are selected for experimental comparison, report must 
limit the generalizability of the findings; cost differentials must also 
be computed as part of the criterion measures. 

4. Criterion measures must be criterion-referenced, 

5. Time intended for experiment must be modest, but of a duration which 
the experimenter can justify as permitting sufficient opportunity for 
treatment effects to be shown, 

6. Sample size necessary to show desired effects must be experimentally 
determined (See Hays, 1963, p. 204), 

7, Experiments should be conducted only in those situations where pro- 
gram performance is sufficiently deficient to permit the experimental 
effects to be shown; that is, products which result in performance of 
80% or more should probably not be vehicles for experimentation 
unless particular sub-populations of learners consistently experiencing 
difficulty are selected as the subjects of the experiment. 



Prototype Tryout 

25. Objective . To prepare a hypothetical plan for prototype tryout including numbers 
and descriptions of subjects, conditions of tryout, justification for segment selected 
to be tryout, categories of data to be obtained and plan for reporting the results of 
the iryout. 

Measurement Procedure . 

2 , Objective will be assessed by 

1. Providing trainees with a thorough description of a product prototype, 
and asking them to plan a tryout, 

2. Inspecting the prototype test plan used in the criterion development 
task. 

B. Criteria 

1. Selection of sample for tryout must be related to population for which 
the product is intended; it cannot be based on an "available subjects" 
mentality. 

2 . Segments selected for tryout must be justified in terms of the par- 
ticular classes of information the developer whishes. They should 
represent major Instructional strategies and should provide sufficient 
time for an adequate sample of various activities to be obtained. 

3. Plan should include forms for recording the categories of data of 
interest to the developer among the following; (a) response data to 
program stimuli; (b) latencies for response data (unobtrusively ob- 
tained); (c) crjteriun performance; (d) time to criterion; (c) inter- 
active responses with learners; (f) observation of learners responses 
for signs of confusion, boredom, etc.; (g) observation of program 
procedures, i,e. # the extent to which the program operates as intended, 
including particular attributes which the developer will attend; (h) affec- 
tive reports. 
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*26. Objective . To be able to conduct prototype testing sessions under conditions which 
call for the reliable accumulation of some of the following calsses of data.- (a) 
response data to stimuli within *.he product; <b) latencies for response data; (c) cri- 
terion performance; (d) time to criterion; (e) interactive responses with learners; 
(f) observation of learners responses for signs of boredom, confusion, etc.: (g) 
observation of program procedures, i.e., the extent to which the program operates 
as intended. 

Measurement Procedure . 

A. Objective will be assessed by 

1. Presenting a videotaped sequence of an instructional tryout to the 

trainee and asking him to collect the data specified. 

2. Observing that the data he intended to collect and the type of data 

actually collected in the prototype trials are in agreement. 

D. Criteria 

1. Trainees will be able to prepare and if necessary, pre-try forms 
which can be used for the reliable recording of data. 

2. In measurement A 1 above, all instances of the data required will be 
recorded by the trainee. 

3. Anecdotal and subjective as well as quantitative and objective data 

will be collected. 

27. Objective. To be able to display prototype tryout data in arrays of special use to 
development personnel, e.g., responses arrayed by stimulus, e.g., frame criterion 
test item and/or objective: lists of incorrect answers produced or selected, or- 

ganization of responses by instructional medium, e.g., errors attributed to instruc- 
tion by tapes, by teacher, by booklets. 

Measurement Procedure. 



A. Objective will be assessed by 

1. Providing the learner with hypothetical or real data obtained in proto- 
type or product tryout, and asking him to summarize the data according 
to specified dimensions. 

2. Inspecting the report of tryouts conducted in association with the 
trainee's criterion development project. 

B. Criteria 

1. All discrete dimensions for organizing data, if pertinent to revision, 
will be included. 

2. ^ata which may be attributed to combinations of pro 3 r in components 
will be so indicated. 

28. O bjective . Given sets of hypothetical or actual data from prototype test, and 

sample instructional segments, to suggest explanations tor obtained data, e.g., sub- 
ordinate task is missing, irrelevant information is included (see Gagn£ and Paradise, 
1961; Silberman, et al., I960. 



o 

ERIC 



36 



8G 



M e £ s uremenc Pr ocedure. 



A. Objective will be measured by 

1. Providing simulated data displays, objectives and instructional seg- 
ments and asking students to justify why a particular hypothesis, e.g., 
irrelevant information is interfering, is or is not plausible. 

2. Providing simulated data displays, objectives and instructional seg- 
ments and, asking trainees to suggest and justify plausible alternative 
hypotheses to explain the data. 

3. Inspecting hypotheses ventured and justified in the trainee’s criterion 
development task. 

D. Criteria 

1. Explanations should center upon instructional dimensions, i.e., learning 
principles and teacher-mediation problems. Hypotheses related to 
media changes will have to be firmly supported. 

2. Different alternatives will be used to explain situations where en route 
objective achievement is high but criterion performance is low from 
cases where both en route and criterion performance are low. 

3. Trainee k will attend and relate in their discussion the relationship 
between cognitive and affective indicators. 

4 . In cases where criterion performance is satisfactory, alternatives to 
shorten, or otherwise reduce the cost of instruction should be con- 
sidered. 



*29. Objective . To revise prototype of products taking explicit account of the following 
inputs; 

(a) response and criterion data on cognitive goals 

(b) attitude indicators, both by observation and self-report 

(c) subject matter critic©' comments 

(d) real world teacbc.s' criticisms 

(e) learners' comments to test supervisor 

(f) literary or aesthetic judgments of style, e.g., rhetoric, camera angle 
Measurement Procedure . 

A. Objective will be assessed by 

1. Presenting trainees with instructional segments, data, and objectives 
and asking them to revise consistent with the above diniens ; ons, 

2. Inspecting the revisions made in their criterion development project. 
D. Criteria 

1. Judges must agree that revisions are consonant with da^a provided. 

2. Where the trainee wishes tc ignore criticisms in revision provided by 
us rs, subject matter critics, or media specialists, he must do so 
explicitly, explaining why he did not take account of selected criticisms. 
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3. To the extent that he can, the trainee will be guided by the principle of 
economy in his revisions; he will never increase the overall instruc- 
tional time by more than 20% as a function of a single tiiaJ -revision 
combination. 

30. Objective. To indicate either in writing or in an interview the revisions implied by 
empirical tryout, in terms of product goals, expected criterion levels, instructional 
sequencing, teacher variability, if pertinent, principles of instruction used, cost, 
product format, content population for which product was intended. 

Measurement Procedure . 

A. Objective will be measured by 

1. Presenting the learner with a simulated product to revise, and asking 
him to prepare a revision plan. 

2. Inspecting the criterion development product produced by the trainee. 

B. Criceria 

1. Papers must include explicit reference to each of the points in the 
above objective. 

2. Revision plan must be articulated; that is, how will revisions be planned 
to take place, what principles will be emphasized, etc. 



31. Objective. To produce a coordinating planfor diverse elements of a complex instruc- 
TTonal "program, e.g., films, lesson workbooks, computer management procedures, 
including a schedule which integrates these into the user period permitted. 

Measurement Procedure. 

A. Objective will be assessed by 

1. Presenting trainee with multiple-component programs to sequence and 
a given instructional time and organizational format, e.g., large group 
instruction, and requiring the student to produce a schedule which 
integrates all components. 

2. In a prospectus for a complex program asking trainees to provide 
such schedules. 

B. Criteria 

1. A schedule must include all components provided. 

2. When more than one component is to be used during a single instruc- 
tional period, e.g., class, the order of use must be given. 

3. In all cases, anticipated time necessary to use any segment should be 
estimated, 

32. Objective. To produce a statement of staff allocation estimates, including required 
tasks, man-hours necessary, and job qualifications of personnel for a complex 
development project. 
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Measurement Procedure. 



A. Objective will be assessed by 

1. In a prospectus asking students to provide such management estimates. 

B. Criteria 

1. job qualifications will be explicitly stated in terms of particular skills. 

2. Man-hours \ ill be analyzed by job classification involved, e.g., 40 
hours for research assistant. 

3. Tasks will be stated both in terms of outcomes desired, e.g., an 
improved instructional program, but in terms of requisite interim 
activities, e.g., conduct prototype test session. 

33. Objective . To be able to plan sessions to provide in-service training for ca-workers 
in selected development skills. 

Measurement Procedure. 

A. Objective will be assessed by 

1. Asking the trainee to prepare a plan for a short training workshop on 
general development skills. 

2. Asking the trainee to choose among the development activities, e.g., 
specification, revision, and to plan a short two-day training program 
on the topic. 

B. Criteria 

1. instructional plans must include specific objectives, plan for criterion 
measures, opportunity for trainees to practice desired skills. 

2 Scope of training must be commensurate with alloted time. 

♦34. O bjective . To be able to prepare a detailed prospectus for a major instructional 
product. 

Measurement Procedure. 



A. Objective will be assessed by 

1. Requiring trainee during the latter phases of his program tn submit a 
prospectus for a major instructional product in his area of com- 
petence. 

B. Criteria 

l. The following major development activities should be included: 

{a) formulation 

(b) specification 

(c) objectives analysis 

(d) test development and tryout 

(e) prototype development 

(f ) tryout and revision 

\g) development management 
<h) field test 
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2. Activities such as formulation and specification should be fully ex- 
panded. Justification for product must be provided. Procedures for the 
solicitation of review of formulation and specification must be provided. 

3. Objectives must include a consideration affective outcomes. 

4. Sample analysis of a complex objective is required. 

5. Basis for initial media selection, including print, must be provided. 

6. Description should be provided of subjects, numbers, conditions, under 
which prototypes are to be tested. 

7. Types of data relevant for revision should be included. 

8. Basis for evaluation should be described. 

9. Sample test items should be included. 

10. Estimation need for teacher training, user orientation, etc., should 
be included. 



11. To include a planned schedule of events for activities inch; Ld in the 
prospectus. 

35. Objective . To prepare detailed forms for use in soliciting e ternal review, e.g., 
users, subject matter experts, to obtain critiques following specifications, proto- 
type development, etc. 



Meas ure ment P roced ure. 

A. Objective will be assessed by 

1. Presenting classes of information desired at various points, e.g., in- 
formation for external review by users to trainees and requiring that 
they prepare forms which solicit such information. 

2. Inspecting forms produced for the criterion development product. 

B. Criteria 

1. Both selected and constructed responses may be solicited. 

2. Language of form should be easily comprehensible to respondents, 
i.e., tecbnocratese is not permissible. 

3. If preferences are solicited the form must provide anchor referents, 
i.e., preferred to what? 

4. Scaling decision must be justified. 

5. Form must be brief. 



Product Assessment 

In addition to the foregoing competencies, trainees would be expected to achieve 
certain objectives associated with how one evaluates a completed product, or at least an 
early version of such product. Accordingly, the following objectives from the set pre- 
pared for product evaluators would also be sought for the product developers: Numbers 
3, 4, 7, 8, and 9. 
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Competencies for the Practlcum Experience {or Product Developers 



The following competencies apply particularly to the practicum phase of our training 
efforts for product developers: 

I. Vocobulary Comprehension 

The necessity f or skillful communication, both in terms of being able to express 
oneself effectively ro understand accurately communications from others, is 
critical for the future product developer. The trainee should be able to express 
himself to others using non-technical words in a fairly precise fashion and yet the 
trainee should understand both oral and written communication of a technical and 
slang nature in each of several fields. 

A. Competency 

The trainee will identify the meaning of key technical terms in fields related 
to educational product development by selecting the correct synonym of the 
term, one of the referents of the term, or an appropriate inference of the 
term as used in a sentence or paragraph. 

B. Test Generating Procedure 

For each of the agencies, prepare a list of 50 to 200 vords which constitute 
the key terms for product development to understand. Have ihe list reviewed 
with deletions and additions by persons in the agencies who are recognized as 
capable judges. Make up a glossary of this list for each agency, containing 
between 200 and 500 words (this glossary might be used by the trainee for study 
purposes). Using a sampling procedure stratified by agencies, prepare a list 
of words to be used on the criterion test, the number being ten percent of r he 
total list. 

1 For half of the terms selected, piepare a vocabulary of multiple-choice 
items in which the word by itself appears in the stem, The five alternatives 
will constitute either synonyms for the word or characteristics (physical 
attributes, functions, or relationships). 

2. For the other half of the list of words selected, incorporate them into a 
sentence or a paragraph. Then pose multiple-choice questions which require 
the trainee to identify an appropriate restatement of the sentence or a 
logical inference from the sentence. 

C. See .Sample Test Items 



Sample Test Items for Vocobulary Comprehension 



Fart of this test could consist of single word stems. 

A gaffe r is concerned with: 

l. sound 
•2. lighting 

3. script 

4. art work 

5. film processing 

B. Sentences and Paragraphs 

Instead of simply presenting a single word in the test-item stem, the term could 
be incorporated into a sentence: 



A. Words 



3 



ERIC 



41 




"Emeryville High School has introduced the pontoon system in tenth grade History 
and English." From this statement you could infer that in the tenth grade at Emery- 
ville High: 

1. English is a prerequisite to history. 

2. Classes in these subjects have been reduced in size. 

3. Several tracks, for different ability levels, have been established. 

4. All teachers of these subjects meet regularly as a small group to plan 
their units. 

*5. For many students, classes in English and History are scheduled one right 
after the other each day. 

SAMPLE 2 

During the shooting of a scene tor an educational film, you hear someone say, 
"We need a gobo." You may infer that: 

1. The camera needs a zoom lens. 

2. The sound equipment isn't working. 

*3. The lighting is too intense. 

4. A change in background is desirable. 

5. Someone is needed to move the props* 

II. Technical Information About Production 

A good product developer must have a certain level of information about an educa- 
tional product in terms of technical procedures within an agency. Although this in- 
formation is only at an introductory level, it should enable the trainee to relate 
effectively to many persons with whom he will cooperate In product development in 
the field. This technical background will require vocabulary treated in the previous 
section. 



A. Competency 

When presented with a major technical task of physically producing the printed 
material, Lhe artwork, the film, or the sound recording, the trainee will identify 
pr< cedures most likely to be effective. A major technical task refers to a part 
of the production process so central that product developers must be able to 
describe the procedures involved; without such information, the trainees will 
be demonstrably less effective in the design and collaborative roles in material 
production. 



B. Test Generating Procedures 

Select two judges to prepare a list of major technical tasks as defined above, 
one judge being a technician with considerable experience In the field, the other 
a product developer. Randomly select five of these tasks for the criterion test. 
For each task pose a problem within the realistic operations of the company 
(e.g., a sales report, a memo from the company president, a complaint from a 
user, a staff review). Fresent the problem in a paragraph followed by one or 
more multiple-choice items which require the selection of {!) the best state- 
ment of what the problem is or (2; the best procedure to adopt. Both judges 
must agree that the item in final form meets the criterion posed above. 



C. Sample Test Item 

See Item No. 2 under Sample Test for Competency 5. 
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III. 



Information About Use of Products 



As a result of his internship experiences, the trainee should have an intimate 
knowledge of the Eetting and the problems in which instructional products might be 
used. He will have an adequate opportunity to learn about the decisions which must 
be made before instructional products are ever put to use. He must understand the 
conditions under which products are used in an educational setting be it a classroom 
or on the job training situation. Such information should equip him to plan and re- 
vise more realistically in his product development work. 

A. Competency 

1. When shown a training problem in a particular user setting (home, school, 
industry, or government) calling for the use of an instructional product, 
the trainee will describe the nature of the dicisions to be made by an 
agency in seeking and selecting a product, the criteria used in making such 
decisions, and the facilitating and constraining conditions under which the 
product is likely to be used. 

2. When presented with a completed instructional product, or a plan for the 
creation of a product, the trainee will identify needed revisions, if any, 
necessary for ultimate adoption and optimum use of the product; in order 
to show why the revision is necessary, he will describe the characteristics 
of the setting to indicate why the product is likely to be rejected for adop- 
tion or why the product will fail to bring about expected outcomes, 



D. Test Generating Procedures 

For each kind of agency in which products might be used, formulate a list of 
the d ecisions to be made by the agency in adopting or continuing the use of the 
product. For each decision list the criteria (the kind of considerations used in 
making the decision) which will probably be used by the decision agent. For 
each agency list the conditions under which the product will be used which must 
result In constraints upon the product and which offer promising alternatives 
for improved product use. 

(’von'olfl frtr Cn>>iwr»f. 

Declsions to approve or continue products are made by the School Board, 
superintendent, principal, curriculum supervisor, or teacher depending on the 
scope and cost involved in the change. 

Criteria for administrative decisions include questions of public relations 
(the public’s reaction toward the decision), cost of the product, administrative 
cost, schedules and space constraints, amount of teacher training required, 
and teacher reactions. Teachers criteria emphasize appropriate and significant 
teacher’s role (neither a puiely mechanical, clerical one, nor a threatening 
overly-difficult role), value of the outcomes, provision for Individual differences, 
no mcrease in teacher load, reasonable amount of change in schedule and class- 
room procedures, and expectations about accountability. 

Conditions of use Include group settings, restricted space, competing 
demands for other curriculum activ : ties, possible group facilitation (through 
use of tutors), logistic constraints of material and equipment storage (when 
not in use), and possible use of parents or community personnel for home or 
school assistance. 

J. To develop measures of Competency (1) above, develop a number of essay 

questions as follows: 
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a. Formulate a sot of criteria by which situations in home, school, in- 
dustry, or government may be identified where the most critical needs 
for products exist. 

b. Using these criteria, formulate a variety of situations where a demand 
for a product is explicitly or implicitly indicated. 

c. In the first part of each essay question, describe in sufficient detail 
one of these situations to make it an example typical of a large number 
cf situations which might be encountered. 

d. In the second part of the question, pose one or more of the following 
kinds of questions: 

(1) What individuals are likely to make decisions about this need for 
products? 

(2) What are the important criteria which will be used in making these 
decisions if a product is involved? 

(3) If a product is developed to meet this need, what will be the features 
of the instructional situation which will force constraints upon the 
way the product is designed or presented? What are the options 
which the product developer might have in designing the product for 
this situation? 

e. The scoring system for the answers should pay particular attention to 
the way in which the trainee identifies the practical factors of adoption 
and use of the product in relation to the list of decisions, criteria, and 
conditions previously prepared aj indicated above. 

2. In creating measures of Competency (2) above, develop a number of essay 

questions as follows: 

a. Create a list of the features of products which are likely to constitute 
a deficiency cither because the product would never be adopted for use 
or because it is unlikely that it can be put to use under typical educa- 
tional or training conditions. 

b. Formulate a description of a producr (preferably an existing one) in 
which one or more of the above deficiencies is present. 

c. Include this description, as the first part of the essay question, along 
with an actual sample of the product {including the manual or other 
orientation materials). 

d. In the latter part of the question, ask the student to identify the changes 
needed in the product, if any. Ask him also to show the basis for his 
decision in terms of the field conditions in which the product might 
be used. 

e. Score answers as in Competency 1 stressing, of course, the relation- 
ship between the product features and the list previously prepared. 

For the section dealing with a plan for the creation of a product, adapt the 

above procedure as follows: 

a. Formulate a description of a plan for the development of a product in 
which inadequate provision has been made to guard against the deficien- 
cies in the previous list. 

b. Ask the trainee to identify the missing or inappropriate steps in this 
plan and to Indicate why his proposed revisions are necessary. 
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Sample Te t T tem 



You are employed as a product developer by a publishing company. You have just 
received the following memo from the assistant editor. The note is accompanied by a 
draft in manuscript form about two paragraphs long as indicated. 



Memo To: Product Developer Smith 

From: Assistant Editor Jones 

"Please review the enclosed draft of a section of the teacher's manual 
for our Kindergarten Learning to Think Program. Send your comments 
for possible revision to me. It isn't necessary to rewrite the stuff. Just 
let me know what you think about it." 



DRAFT 



Page 5 



Teacher's Manual 
Learning ta Think 
The Learning to Think Program provides a highly replicable 
method of attaining the behavioral goals of the unit. In repeated 
tests, where all sources of variability were brought under a 
high degree of control, the performance of children in the 
specified population has consistently met the 90-90 criterion 
on the post-test. 



One excellent feature of the program, therefore, is the way 
in which the administration of the unit has been brought under 
a rigorous level of control. All communicationsof the teacher 
are carefully standardized in terms of both wording and timing. 
Children's responses to each verbal stimulus are recorded in 
the workbook and given immechate reinforcement. It is this 
high degree of control over the classroom setting which pro- 
vides a major advantage in efficient production of the specified 
performance. The teacher is never left in doubt as to what to do. 



Send your reply to the memo by writing about 30 wordo. 



Scorin g key: 



L T:chnical level of presentation too high for teachers 2 points 

2. Flavor of control too repulsive for teathers 2 points 

3. Implied role for teacher is a mechanical one 2 points 

4. Workbooks seen as the major instructional activity 2 points 

5. General style and presentation of the reply 2 points 

Total 10 points 
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IV. Procedurol Infoririot ion Involving Personnel 

A good product developer or evaluator should know a lot about how and where de- 
cisions are made in various industries concerned with instructional products. He 
needs to know a lot of factual information about how such an industry operates. 
An information type test could be developed to see how well the trainee had learned 
to find his way around. 

A. Competency 

When presented with the problem of product development involving other per- 
sonnel for decisions or services, the trainee will indicate who should be con- 
tacted and the nature of the contact. 

B. Test Generating Procedure 

Make a list of the points in the development of products where other persons 
will need to be contacted, (1) within a particular organization concerned with 
product development or (2) with individuals outside the organization. For each 
test item, present the trainee with a description of a project involving the de- 
velopment of a product. At a given point describe a problem which involves a 
necessary contact with other persons (although this latter point is not indicated). 
The trainees response is scored in terms of the adequacy with which he iden- 
tifies perso.. whose judgments, services, or decisions are necessary and the 
adequacy with which he makes it clear why these are important. 

Sa mjjlc Test Item for C om petency A . 

You are ready to have materials tried out with a group of typical third graders for 
?n instructional program. Unfortunately, it is in the middle of July and no private or 
publir schools are "in session. How will you proce d to get subjects? The try-out will 
take about two weeks with six children. 

1, Contact the school principal in the neighborhood to see if enough school 
children might be recruited. 

2. Knlist the cooperation of your fellow employees ir. "rounding up" their 
own children if they are in the third grade. 

3 The Kpiscopal Church has a two-week church school going so contact the 
rector. 

*4. Make an appointment to see the director cf the day care centers, under the 
supervision of the schools. 

5. Talk to a third grade teacher to see if she can get some of the children in 
her last year’s class together. 



V. Ccmnunico'wn with Agency Personnel 

An important aspect of product development is the ability to communicate within 
the atmosphere of many agencies. I rainees frequently have bad social contacts 
limited to academia and are inept in dealing with personnel in the industrial world. 
For this competency an adequate vocabulary, comprehension of words, sentences 
and paragraphs, as mentioned earlier is clearly required. But more is involved; 
one must be aMe to pick up t be nuances of meaning and feeling informally expressed 
in a conversation, a memo, or letter. While the following procedure, proposed as 
a way of defining this outcome, involves only "listening comprehension," it is 
assumed that trainees will also be ahle to express themselves as necessary. 
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A. Competency 



The trainee will identify the affective as well as cognitive meaning expressed 
informally in a communication relating to product development. 



B. Test Generating Procedure 



Identify the kinds of situations where comment carrying an important approach- 
avoidance implications is present, where two coalitions are met: the situation 
is one likely to be encountered by a product developer, and the appropriate 
interpretation will facilitate further relationships in the organization. 



Select several of these situations and for each one prepare either a (1) dialogue 
between two persons where the conversation deals with product development, or 
(2) a written message in the form of a memo or letter which might be received 
by a product developer. 



Exam ples of Dialogue Settings 

E Two school people (a curriculum supervisor and a teacher) discussing 
flexible scheduling, modules, and the platoon system. 

2. Two television personnel, one a program director and the other a tech' 
nician, talking about changes needed in taping a program. 

3. A film production manager discusses with a photographer his evaluation 

of a strip film. , . 

4. A training director for an aircraft company and a plant manager are 
discussing a new in-service training program being prepared. 



Examples of Written Me ssage 

I message from the head of production services asking for a go-ahead with 
a printing job but indicating that he has serious reservations about certain 
technical aspects of the assignment, a reservation which is implicitly 
expressed but should be noted by the trainee, 

2, A copy of a formal letter by the president of the company presumably 
''backing up" the product developer's plans but in reality showing con- 
siderable doubt about the value of the product 
3 A note from the supervisor giving a tentative agenda for a meeting to 
assess progress on the , oduct. The items go casually listed reflect the 
supervisor's own views as to how the plans migi't be changed. 

4. A copy of a letter written by a classroom tea :ner to the principal subtly 
expressing the view that she is being threatened by the way the product is 
being introduced into her classroom. 

Prepare the dialogue for a sound or a TV tape. It may also be presented in 
printed form. The trainee is then presented with a multiple-choice test asking 
him to interpret the dialogue. Printed messages rr.ay be in typewritten or jn 
handwritten form. 



E xamples of Questions 



1. 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 



What did A tell B to do? 

Why d Id B object? 

How did B feel about the situation? 

What compromises was A willing to make? 
W r hat does A think of the product? 

How does A feel about the development plans? 
What problem, if any, is raised by this memo? 
Suppose you sent a letter saying that you 
conseq ices? 



agree; what would be likely 
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Sample Test Item 



Assume that you are standing in the corridor of a film f roduction company talking 
to the production .manager. As a technician walks by, the manager turns to him and says: 

’Hey, Bill, you know the slides we copied yesterday? Well, there was too mu:h 
contrast build-up." 

"1 can’t understand it, chief. We shot it with a strobe." 

"Well that equipment must be bugged. Try something else." 

"O.K. Chief. I’ll take care of it." 

"By the way, Bill, you didn't have any hot spots this time." 

"Well, it r s the fault of that guy Jim, you gave me for a helper. See you later." 
Vocabulary Item 

1. What seerr.s to be the problem in this conversation? 

a. 1 he colors in the picture were washed out. 

b. The objects in the scene were improperly arranged. 

c. There were too many reflections. 

*d. The details in the shadows were not clear, 
e. The colors were not faithful. 



Information Item 

2. What might Bill do about the problem? 

a. Use a .Od neutral J ansity filter. 

•b. Use quartz lights and postfiash the film. 

c. Use lamps with a higher wattage. 

d. Use an approximate diffusion filter. 

e. Use a shorter exposure time. 

In ter pre tation Item 

3. What does Bill think about his helper Jim? 

2 He is using his as a scapegoat, 

*b. He is complimenting him. 
c. He is blaming the "chief" for assigning him. 



Product Evaluation Competencies 

We turn now to the o! jectives for our product evaluation trainees. 

At the conclusion of certain elements of the training program, product evaluation 
trainees will be able to do: 

*1. Object iv, Aid consumer groups to articulate more explicitly the ot jectives they 
wish a given product to accomplish. 
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Measurement Procedure. 



A. Objective will be assessed by 

1. A contrived interview test situation, where the trainee interacts with a 
staff member who is playing the role of an educational consumer. 

2. The staff member-actor will have several precise instructional objec- 
tives in mind, but will present loose, inexplicit goals to the trainee. 

3. The degree to which the trainee can ferret out the measurable instruc- 
tional objectives will provide an index of the trainee's competence. 

B. Criterion 

1. The primary determinant in this simulation situation will be the con- 
gruence of the objectives finally decided upon by the trainee with those 
originally identified by the staff member-actor. (We'll have to sharpen 
this procedure considerably, for chere are obvious difficulties in 
using such a measuring device. Nevertheless, this approach will pro- 
vide a close simulation of what we actually wish product evaluators to 
do when working with educational consumers.) 

2. Objec tive. Infer legitimate measurable instructional objectives from extant educa- 
tional products, such objectives to possess the following attributes whenever possible: 

(a) content generality, (b) criteria of adequacy by which to judge the acceptability 
of a learner’s constructed response (as opposed to selected response), (c) a range 
of cognitive and affective 1- arner behaviors (and psycho.notor, if relevant). 

Measurement Procedure. 



A, Achievement of the objective will be assessed by 

1. Trainees will be given a variety of products and any accompanying 
reference materials, for example, instructor's manuals, then asked to 
infer the objectives from the materials. 

2. The trainee will also be obliged to present a statement defending his 
inferred objectives. 

B. Criteria 

1. The logical consistency and evidence -supported inferences made by 
the trainee will be judged by staff members. 

2 , The attributes possessed by the inferred objectives, as outlined above, 
e.g., content generality, will serve as a second criterion of the attain- 
ment of this objective. 



*3. Objective. Generate a wide range of measurable objectives to assess many and 
d 1 verse types of learner outcomes, e,g., high level cognitive and affective outcomes, 
using both standard and more esoteric measurement tactics, for example, unobtru- 
sive measures. 



M easurement Procedure. 

A. Achievement of the objective will be assessed by 

i. Trainees will be given a series of general, noti-behavioral objectives 
and asked to generate alternatives to assess attainment of the objective. 
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2. Requirements will be supplied in some cases, for example, trainees 
will be asked to supply at least three measures which would be used to 
assess cognitive learner behavior at higher then the lowest level on 
the Bloom taxonomy. 



1. The number of types of student objectives generated as well as then* 
congruence with the posited specifications, e.g., in the objective 
domain and at the prescribed level, will serve as standard for judging 
trainee performance. 



* 4 . Objective. Devise a range of defensible criterion-referenced measures to assess 
the learner's attainment of the diverse objectives associated with number three 
above. Defensibility here implies the competence to employ both standard and very 
recently devised techniques for improving the adequacy, e.g., reliability and vali- 
dity, of the criterion-referenced measures. 



V e asurement Procedure , 

A. Achievement of the objective will be assessed by 

1. Essay tests can be used to assess the familiarity of trainees with 
recent advances in the measurement field, particularly having to do 
with tactics for improving criterion-referenced measurement tech- 
niques. 

2. Given measurable objectives and situational constraints, e.g., time 
available for testing, cost of measurirg procedures, cost of repro- 
ducing measurement devices, etc., the trainee will have to ger.. rate 
and, via both a priori (content validity) and a posteriori (field trials) 
methods, defend the adequacy of the measures devised. 

B. Criteria. 

1. Essay questions can be judged on the basis of the trainee's display of 
current information regarding the development and improvement of 
criterion-reference ter>ts. We will be particularly attentive to his 
ability to discriminate between measurement procedures more suitable 
for norm-referenced measures than for criterion-referenced measures. 

2. The staff's judgment ot the rigorousness and technical adequacy of the 
procedures followed will serve as the prime criterion by which to judge 
this objective In particular, the procedures employed for securing 
content validity and for treating try out data will be scrutinized. The 
trainee's written rationale fo^ his actions will be inspected for cogency. 



*5. Objective . Construct the requisite documents for delineating a domain-referenced 
achievement testing scheme (Hively, 1970) whereby the congruency between objec - 
fives and measuring procedures can be increased. 



Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. This objective will he assessed by having the trainees produce item 
forms according to a domain-referenced achievement testing approach. 
The trainee will be given several measurable objectives and asked to 
produce an item form for each which could guide a test item writer. 
(Test item here is used in the broad sense, signifying more than mere 
paper and pencil measures.) 



B. Criterion 
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B. Criteria 



1 The item forms produced by the trainees can be judged both on internal 
attributes, that is, the judged likelihood that the forms would yield 
homogeneous test items. 

2 . The item forms to be given to item writers who will produce items 
according to those forms, the homogeneity of the items being subse- 
quently verified on the basis of field testing with appropriate learners. 

6. Objective . Devise economically efficient matrix sampling schemes whereby learner 
behaviors resulting from ine use of instructional products can be secured* i.e., 
involving both person sampling and item sampling of learner behaviors. 

Measurement Procedure. 

A. Achievement of the objective will be assessed by 

2. The trainee will be obliged to prepare a written plan for securing learner 
data via a matrix sampling scheme. 

B. Criterion 

1. The technical adequacy of the plan will be the primary criterion for 
judgment, that is, gross flaws in the item sampling procedure, having 
too few or too many to be completed by each learner, etc. 

*7. Objective . Detect unanticipated outcomes which emerge from learners’ use of in- 
structional products. 

Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. This objective would be assessed by providing a variety of real or 
fictitious anecdotal test data associated with the use of an instructional 
product. Embedded in the anecdotal and test data will be negative and 
positive instances other than those associated with the instructional 
objectives for which the product has been selected. The trainee s task 
will be to detect any such anticipated outcomes. 

B. Criterion 

1. The accuracy with which the built-in anticipated outcomes are identi- 
fied will serve as the criterion. 

*8. Objective, Devise plans for gathering learner post-instruction performance data 
so thar~reasonaule inferences can be made regarding the nature of the impact of 
instructional products on the learner’s behavior. Tor instance, trainees would 
have to be conversant with the advantages and limitations (for evaluation) or certain 
research designs such as the interrupted time series design. 

Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. A series of hypothetical situations would be presented to the learner 
whe^'in he is required to recommend an evaluation design which could 
be employed to pr duce the required inferences regarding the instruc 
tional product's effectiveness. Certain elements will be built into the 
description which should incline the knowledgeable trainee to select 
certain designs In preference to others. 
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B. Criterion - 

l. Judiciousness of selection of designs consistent with the factors built 
into the simulated situations, for example, if the pretest measure- 
ment is clearly reactive in nature, the trainee will eschew all variants 
of pretest-post-test designs and will choose, instead, designs such as 
the post-test only control group design. 



* 9 . 



Objective. Conduct analyses of data such that both hypotheses testing and estima- 
tion statistics are yielded, thereby permitting relationships among variables to be 
detected as well as the magnitude of impact produced by the use of the instructio 
product. 



Measurement Procedure . 

A. Achievement of the objective will be assessed by 

1. Fictitious written or oral situations will be presented to the learner 
wherein he is asked to describe the data analysis techniques which 
would be suitable. 

2. These would be constructed response test items, rather than multiple 
choice measures, so that they will approximate real life situations 
more closely. 



B. Criterion 



1. Consonance of data analysis techniques suggested with requirements 
of the fictitious situation, for example, the use of estimation procedures 
when magnitude of impact is desired. 



*10 Objective. Make recommendations regarding adoption of specified instructional 
products ' within a cost/effectiveness context. Effectiveness is to be defined in terms 
of learner growth. Costs refer not only to time requirements, actual financial cost , 
but all other related costs such as teacher morale, public acceptance, etc. 



Meas u r ement Procedure. 



A. 



Achievement of the objective will oe assessed by 



A written examination, in which a variety of data are presented to Lhe 
trainee, both initial expectations of the consumers, objectives of the 
instructional products, field tests results, etc. The task of the trainee 
will b" to write a defensible recommendation, complete with elaborate 
justification, regarding the adoption of the instructional product under 
scrutiny. 

Providing a real product evaluation situation fot the trainee, and having 
him follow a product through its implementation and testing, in order 
that he provide a clear recommendation for the potential user regarding 
the product’s value. 



B. Criterion 



The report which the trainee prepares will serve as the da a to be 
judged regarding the attainment of this objective. Defensibihty of 
decisions based upon data, inclusiveness of criteria employed by the 
trainee, etc., will serve to Judge whether this objective, in essence, 
will constitute the heart of a comprehensive performance test of the 
product evaluator’s skill. 
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1 1 . Objective. Reconcile the disagreements between products designed to attain specific 
goals and" user expectations (goal requirements) which are at some, but not complete, 
variance with the produces intended goals. 



Meas urement F 3 roccdures. 

A. The objective will be assessed by 

1 The trainee wil 1 . be given a fictitious written description of a situation 
where the product's goals and the user r s expectations are in some, 
hut not complete disagreement. 

2. The trainee will be asked to generate a scheme for taking these dis- 
agreements into consideration in order to assist the consumer to make 
a judgment regarding the adequacy of the product. 

B. Criterion 

1 The comprehensive systematization of the scheme used by the trainee 
will serve to indicate whether this objective has been achieved. For 
example, a scheme such as the Program Eair F valuation (Popham, 
1969) would prove serviceable in such instances, as would comparable 
analytical approaches. 



STUDENTS 

As suggested by the nature of our dual duration training programs, wc shall have two 
types of trainees. Some will be with us for an entire calendar year, some for only a week 
or so. Let's examine both types. 

I nna -term trainees . Long-term trainees will either be degree ^candidates or _non- 
degreecandidates. If they wish no degree, either master s or doctor s degrees, then all 
training will take place under the complete supervision of the Consortium. If the trainee 
wishes^ to secure a master's or doctor’s degree, then a closer relationship Will occur 
with the UCLA Graduate School of Education under whose auspices th ^^ re ^f 
granted. The existence of established degree_progranis in both of these 
-namely, instructional- product 'Research* and" tsuucatiunai r.vaiijduob, will render ti.e 
consummation of a degree program by trainees relatively routine. 

The master's degree candidates will normally remain from three to six months 
beyond his calendar year training program to complete thesis rc qu i re me nl s ^ r ses 

which could not be taken because of conflicts witn training \ rogram requireme t . 

The doctor's degree candidate will normally remain one to two years after comple- 
tion of his training program calendar year in order to take remaining course v*ork and 

complete a dissertation, qualifying exams, etc. If the doctoral cand j^ ^rb^vo^d^^the 
gram with a master's degree already, then he would normally take one year beyond the 

basic calendar year training program. 

Many of the training sequences for either product evaluators or product developers 
will be taught either by UCLA faculty or by competent professionals from consortium 
agencies who can be readily certified to offer regular coursework, 

The long-term trainees will either choose the training program because one of its 
two phases interests them, much as a student now chooses a field u “enable 
because their organization wishes to send them to receive training which will enable 
them to function more efficiently in that organization, We anticipate, for example, that 
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certain regional laboratories or commercial publishers will send some of their personnel 
to complete the long-term product development training sequences. Publishers and large 
school districts or state departments of education may send individuals wishing to com- 
plete the product evaluation training sequences. 

Basic Criteria for Selection . In general, the following criteria will be employed in 
the selection of trainees. 

A. A baccalaureate degree from an accredited institution; or the submission of 
evidence of productivity in any educational field. 

B. A command of written English to be judged by a lengthy statement required in 
the application as well as by inspection of written work submitted, if any. 

Specific preferences will be given to applicants as follows: 

A. Applicants presently employed in development agencies. 

Applicants with job responsibilities directly related to development or product 
evaluation will be given preference. One explanation is that such individuals 
would be able to make use of their training immediately upon completion of 
their program. Trainees would also be encouraged to set up in-service training 
programs and thus multiply the effects of the consortium training. 

Representative of classes of individuals to be given preference (not in listed 
order) are as follows; 

1. Private developers, e.g,, publishing houses 

2. School district personnel 

3. Staffs of R and D centers and regional laboratories 

4. Staffs of Title I and 111 projects 

B . Mj n ority group members. 

Because the Consortium staff believes that members of mb.--* ,, t y 

bring special insight to instructional problems for minority ... . vs, 

and in view of the key social significance of today’s minority g: - ,a I 

problems, applicants who are minority group members will K c xe 

whenever other factors are relatively equal. 

C. N eeded specializations. 

Applicants who intend to focus on areas of high priority w * = ned 

favorably, examples of such areas are urban instructional setnr. rn, 

vocational education, medical development, reading, and evalu r 

D. S upervisory role. 

Applicants whose present employment tequlres the super vB ,ncl 

would he given preference if their application indicates that d ve a 

relatively expanded sphere of influence. 

H. Age. 

Given equal qualifications, preference will be given to y -,r^ 

assuming they are likely to have a longer period of produ i v he 

more adaptable. 
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F. Media Experience. 

Computer programmers, filmmakers, or other technical specialists, who apply 
will be given preference, 

G. Intellectual Prowess . 

If Graduate Record Examination or College Board scores are submitted, 
preference will be given to high scores. However, a modest score, e.g., 1,000 
on the GRE, will not exclude the applicant, should be present other evidence, 
e.g,, prior creative work. 

Stipends . For a specific number of long-term trainees who are not already receiving 
salaries from the institutions sending them, such as a regional laboratory employee 
who continues to receive his normal salary from the laboratory, stipends will be pro- 
vided. At the outset, until the Consortium training program becomes well known as a 
program where competencies are acquired and, thereafter one can secure high level 
professional employment, these stipends will have to be more numerous, As the training 
program's high quality is recognized, fewer of these stipends will be needed. Indeed, 
there has been some consideration of charging tuition fees after the program is well 
established, thereby defraying costs. 

The stipend will be $2,400 to $2,800 per calendar year plus $500 per dependent and 
a one-way travel allowance from home to the training program of 10 certs per mile. 
For non-degree candidates, only one year's stipend is available. For a master's degree 
candidate up to ore and one half years' stipends are available. For a doctoral candidate 
up to three years' stipends are available. The number of stipends requested per year 
will have to be used judiciously by the Consortium training staff, that is, it will be to 
their advantage to expedite the progress of degree candiaates through our program so 
i* at other trainees can use the stipends. 

Because of the heavier emphasis on development training in our program, we are 
requesting ten stipends for product developers and five stipends for product evaluators. 
As we anticipate that about half of our trainees will not be degree candidates, sonic of 
the stipends will be used for those trainees as well as for degree candidates. We expect 
20 full-time development trainees the first year and 10 full-time evaluation trainees 
during that year. 

Short-term trainees. In numbers alone we anticipate reaching far more individuals 
through our short-term training programs. Most of these will be carried out at the Train- 
ing Laboratory or on site at a consortium member's locale. Some will be conducted at 
different points throughout the U.S., in response to particular training requirements. 
We will definitely develop the capacity during the first year to take a one to two weeks 
staff Into the field to provide the initial elements of a training program which, by careful 
follow-up via correspondence and t lephone consultation, can be extended for up to one 
year. These short-term institutes would be held in locales particularly in need of training. 
For example, a delegation of black educational researchers recently submitted a request 
to the AFRA Task Force on the Training of Fducational Researchers that a series of 
special training activities be im de available for black educational researchers in the South 
who, because of local situations, could not secure such training. We will take a short- 
term training program into that geographic region to prepare product developers and/or 
product evaluators depending upon the need as evidenced by advanced proselyting efforts. 

Most of the short-term sequences would be of a general nature. Follow-up readings, 
instructional programs, and telephone -correspondence interaction w^ould be used to 
increase the potency of the instruction. Other short sessions w r ou!d be highly parti - 
cularized, dealing with special problems, for example, the use cf learner performance 
data in the revision of products or the construction of objectives and measures for pub- 
lished materials which possess none- Often, these short-term sessions will be solicited 
by particular organizations, e.g., an R and D Center. 
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We anticipate reaching between 200-300 trainees the first year through these short- 
term efforts, some of which may last as long as several months. During the second 
year of the program's operation we should reach between 300 and 400 individuals. Seme 
of the individuals trained in the short sessions, we surmise, will wish to undertake a 
long-term training program at a later point. 



ADMINISTRATION OF THE TRAINING PROGRAM 

While for financial purposes UCLA will be the prime contractor, the Consortium 
training program will be administered by the full-time staff of the training program 
through the training program director as approved by a governing board constituted by 
one designated representative of each Consortium agency. All major policies guiding 
the Consortium must be democratically approved by this governing board. The director 
and the full-time staff (or nearly full-time) will implement these policies. 

The governing board will meet for half-day sessions once every two months on a 
regular basis but all board members will also be invited to staff meetings on alternate 
months for informational purposes. Special governing board meetings may be called by 
the elected chairman of the governing board with the approval of at least one-half of 
the board members. A token honorarium of $50 will be given to each governing board 
member for his attendance at each meeting. 

Working relationships between the Consortium member agencies and the Consortium 
training staff will be coordinated by the full-time Consortium staff under policies approved 
by the governing board. 



EVALUATION 

Because of the results-orientation guiding this program, both in terms of its pro- 
ducts as well as its process, the evaluation of the program will be comprehensive and 
rather straightforward. 

For both the product development training sequences and the product evaluation 
training sequences we shall systematically monitor the ability of the trainees to dis- 
play the desired skills, Measurement of the long-term trainees will occur every three 
months on a matrix sampling basis whereby we shall randomly sample both trainees 
and objectives (thereby reducing the personal threat of the measurement since different 
trainees will be completing different test measures). Results of this quarterly measure- 
ment will be summarized and transmitted to members of the governing board, all staff, 
all trainees, and appropriate USOH officials. 

Systematic measurement of the results of short-term training sequences will also 
be undertaken, typically at the close of such sessions, although perhaps earlier for the 
more extended of the short-term activities. 

P erformance Tes ts. In addition to the use of the measurement devices related to 
each objective, we shall develop some brand new performance test measures to be 
used with both product developers and rrcduct evaluators. The essence of each perform- 
ance test will be that it pro\ides a simulated opportunity for the trainee to perform, 
generally, the competencies he has supposedly learned. 

1 or the product development performance tests we shall develop several versions 
of a test which presents to the developer the task of devising a replicable instructional 
F^qucrre to accomplish p*especified instructional objectives. Constraints will be present 
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regarding available instructional time, cost of materials, etc,, but the trainee will 
otherwise be free to devise a product that accomplishes the objectives. We shall observe 
which of the alternative development tactics each trainee employs, for these will be 
useful in modifying our program, but the key index of whether the trainee has passed 
the test will emerge when the product is tried out with learners drawn randomly from 
a group of eligible subjects. The product, taking no more than 40 minutes of learner time, 
will be field tested with approximately a dozen such learners and pre- and post-instruc- 
rion scores on measures based onthe objective will reveal the success of the product. 
As time goes by we can begin to establish defensible norms for these tests so that each 
learner's skill can ultimately be contrasted with some reasonable criterion. As indicated 
above, several different performance tests for different subject matters and for different 
type learners will be developed. 

For the product evaluators a different sort of performance test strategy will be 
implemented. Sets of simulated problems in evaluating extant (fictitious) instructio\al 
materials will be prepared for the trainee along with a host of data, some relevant, some 
irrelevant. The trainee will be required to prepare an evaluation report and a set of 
recommendations regarding the fictitious product. These reports will be carefully 
evaluated, using well delineated jury techniques, first by member s of the training program 
staff, then by outside evaluation specialists. As with the performance tests in development, 
we will be able to develop norms for such tests over a period of several years. 

Results of performance tests for product developers and evaluators will be reported 
at the close of the training programs for which they are used. Generally, they will be 
used onlv for long-term training programs, but there are some instances in which they 
could be used for short-term training efforts. 

In addition to trainee results on standard examinations and performance tests, the 
training program staff will also assemble pertinent data regarding such factors as the 
number of trainees completing the program, positions secured after training, etc. 

Systematic Study of Procedures Used by Trainees. The availability of performance 
tests for product developers and evaluators will permit the consortium staff to undertake 
the systematic analysis of the technique ■ used by developers and evaluators ir. their 
efforts to satisfy the tasks set by the performance tests. Particularly with respect ;o 
the performance tests for developers, where pupil performance on measu es (after using 
the newly developed product) will serve as the criterion, we will be able to design small 
scale correlational and experimental studies in order to identify whether certain prac- 
tices, that is, development tactics, yield greater learner growth. By designing these 
studies in such a way that learner entry behavior is controlled, we can discern, for 
example, whether developers who have been directed to prepared products according to 
specified procedu r es actually produce superior materials 

Fortunately, two members of the consortium staff, John McNeil and James Popham, 
have been developing and studying this type of performance test for the past five years. 
Their experience will be invaluable in designing peffoimance tests which satisfy both 
criterion-referenced and norm-referenced measurement standards, that i\ the tests 
must be totally congruent with the task (objective) given the developer but y;: must be 
capable of producing variant scores among those rested. McNeil and Foph.m have re- 
cently synthesized their thinking on this relatively unused approach to proficiency 
assessment in the'r chapter on teacher competence assessment in the revision of the 
Handbook of Research on Teaching (1972). 
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STAFF 



Visiting Faculty 

Not only does our consortium possess a continuing staff of inordinate skill, but we 
have also persuaded the following outstanding educational leaders to participate as 
visiting faculty whose contributions would range from as little as one or two days a year 
to perhaps several weeks or months. Each of the following people, in response to a per- 
sonal letter or telephone conversation, have indicated their willingness to explore details 
cf a visiting faculty relationship with the consortium: 

Fred N* Kerlinger, New York University 

Arthur A. Lumsdaine, University of Washington 

Susan M, Markle, University of Illinois, Chicago 

Jason Millman, Cornell University 

F rnrt Z. Rothkopf, Dell Telephone Laboratories 

Robert E. Stake, University of Illinois 

Patrick f ippes, Stanford University 

Robert M.W. Travers, Western Michigan University 

Ralph W. Tyler, Chicago, Illinois 



Regular Staff 

MARVIN C. ALKIN, Center for the Study of Evaluation. 

Dr. Alkin has been playing an increasingly prominent role during the past two years 
in the nation’s educational evaluation efforts. One recent week alor e, for example, he has 
consulted with the Bureau of Elementary and Secondary Education of USOE as a member 
of a review panel evaluating the Stanford Research Institute’s evaluation of the National 
follow-through Program, and as a special consultant to the New Mexico Legislature 
in evaluating their statewide evaluation program. As director of CSE, his national per- 
spective with respect to a newer conception of educational evaluation as a comprehensive 
aid for decision-makers ranks him with Stufflebeam, Cuba, Provus and Stake as first-line 
evaluation theorists. 

Dr. Alkin is the Director of the Center for the Study of E valuation, University of 
California, Los Angeles, and Associate P rofessor of Educational Administration, Graduate 
School of Education, University of Californ'a, Los Angeles. Hns pru sent professional 
activities include the following: Field Reader for the U.S. Office of Education in the 

areas of Cost-Effectiveness, Educational Finance, Management Information Systems 
and Evaluation of School Systems; Contributing Editor, Educational Technology . 

Dr- Alkin is the author of approximately 30 articles and reports and 20 papers 
presented to professional associations. He is a member of the American Educational 
Research Association, American Association of School Administrators, National Society 
for the Study of Education, and Phi Delta Kappa. 

Dr. Alkin received his B.A. m Mathematics and M.A. in Education from San Jose 
State College. He received h!s Ed.D in Educational Administration from Stanford Uni- 
versity. 
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As Director of CSE, his writings on the subject of educational evaluation have been 
attracting national interest. His expertise in designing training programs for the 
evaluation personnel will be most vital in designing the proposed training scheme. 

DONALD E. ANDERSON, American Tape Duplicators. 

Donald E. Andersu", Vice-President and General Manager of ATD, is expert in 
audio-tape production and duplication. He has a background in recording, editing and 
mastering of educational tapes, as well as a thorough production knowledge of duplica- 
tion packaging and the record keeping involved with the entire process. 

EVA L. BAKER, Graduate School of Education, UCLA. 

Dr. Baker's experience as a member of the SVVRL professional staff, a kay CSE 
staff member, and an associate director of the UCLA Product Research Training Pro- 
gram has qualified her to make unique contributions to the theoretical and practical 
aspects of the projected training program. She has already contributed the chapter on 
educational product development for Macmillan’s Encyclopedia of Education and is 
currently writing the major chapter on educational development for the second edition 
of The Handbook of Research on Teaching . 

Dr, Baker is an Assistant Professor at the UCLA Graduate School of Education, 
and a Co-director of the Instructional Objectives Exchange, a nonprofit educational 
corporation. She received her B.A. in Elgtish and M.A. and Ed.D. in Education from the 
University of California, Los Angeles. She is chairman of the graduate program in In- 
structional Product Research. 

Her teaching interests cover courses in instruction, instructional product research, 
and teacher education. Research training includes the following: Southwest Regional 
Laboratory Summer Training Sessions for Educational Researchers, Summer 19o7; 
American Educational Research Association Presession Staff on Instructional Product 
Development, 1968; California Educational Research Association Staff on Instructional 
Product Development, 1968; California Educational Research Association Chairman of 
Presessions, 1969; Southwest Regional Laboratory for Education*" Research and De- 
velopment, Staff Training Element; Peace Corps Training Teacher Preparation Director, 
1968; Staff, UCTA Center for the Study of Evaluation; Professional Staff, Southwest 
Regional Laboratory for Educational Research and Development, 1967-1969. She has 
produced over 50 articles, reports, and papers and she is current recipient of a 
materials development grant to prepare an instructional product to train educational 
developers. 

G RANT R. CARY, BFA Educational Media. 

Thirteen years experience in the Los Angeles City Schools as a teacher of Science 
and Mathematics, television consultant, Adult Education teacher and consultant, Audio- 
Visual Specialist; nine years experience as a Junior College Instructor, Evening Divi- 
sion; Coordinator for Adult Education television programs produced at KABC Television; 
co -founder and partner, Si^ma Educational Films, Assistant Director of Product Develop- 
ment, BE A Educational Media. 

JOHN E. COULSON , System Development Corporation. 

Dr. John E. Coulson, a Senior Research Leader at System Development Corporation, 
is Assistant Manager of the Education Systems Department In this position. Dr. Coulson 
supervises and participates in a wide rarge of projects involving the development of 
educ ational technology and the application of this technology in operational school systems. 
Dr. Coulson is project leader of the Head Start Revaluation that SDC is conducting for the 
Office of Child Development of the U.S. Department of Health, Education and Welfare. 
This is very similar to the work herein proposed in that it is a national analysis of 
the impact of Head Start programs onoi ,f ^rent children and their families and involves 
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detailed compute, manipulation of data. His activities since joining SDC\ c education 
effort in 1959 have included studies of programmed instruction and computer -assisted 
instruction: design of computer-besed laboratory facilities for the study of complex 
interactions in the teaching/learning process; application of tutorial techniques for 
enipir::al development of lesson material 5: and development of criterion-referenced 
diagnostic instruments. 

Di. Coulson holds B.S. and M.A. degrees from the University of Arizona, and a Ph.D. 
in psychology from Columbia University. He was a Lecturer in the School of Education 
at UCLA in 1965-66. He is a consultant to the Air Training Command, and a member cf 
the Review Panel for the Los Angeles City School District for Title 111 (NDEA) proposals. 
He is a Fellow of the American Psychological Association, and a member of the American 
Educational Research Association, the National Committee for Measurement in Educa- 
tion, and the American Association for the Advancement of Science. 

ROBERT T. riLEP , Institute for Educational Development. 

Robert T. Hlep is Vice President of the Institute for Educational Development where 
he is responsible for the operatior of the Western Regional Office and major studies. 
These currently include an assessment of ten years of Title VII-NDEA research and 
dissemination of educational media foi the U.S. Office of Education; and development of 
a cost-effectiveness model for comparison of CAI, Pi, and Traditional Instruction for 
the Bureau of Naval Personnel. 

His Ph.D. degree in Education was gr -nted by the University of Southern California 
where he majored in Instructional techntdogy and psychology, investigating learner 
cna racteristics as related to various programmed media. He received his B.A. degree 
in Education from Rutgers University and an M.A. degree in Psychology from Columbia 
University. 



Having graduated from Rutgers in 1953, he then taught general science at Teaneck 
High School, Teaneck, New Jersey, until 1956, and also served on active duty as an 
officer in the U.S. Air Force during this period. Following completion of his Master 's 
degree in 1957, Dr. Eilep was appointed Assistant Director of Admissions at Renssalaer 
Polytechnic Institute, Troy, New York, with additional responsibilities for the advanced 
program. In 1959, he was appointed Dean of Admissions and Financial Aid at Mills College 
of Education in New York City. He lacerbecame Secretary of the Center for Programmed 
Instruction, Inc., New York City, where his principal duties were Director of the Infor- 
mation and Training Division, Editor of the Bulletin, Programmed Instruction, and 
researcher responsible for examining student interactions with programmed instruc- 
tion, In 1963, Dr. Filep received an appointment as Associate Investigator of the Cinema 
Research Division at the University of Southern California, where he taught and periodical- 
ly teaches graduate and undergraduate level courses* 

He is vice president of the Educational Media Council and is currently a member of 
the E DU COM Unteruniversity Communications Council), Task Force on Continuing 
Education, and Advisor to the ERIC Clearinghous on Early Childhood Education. He ha& 
also been a part-time faculty member of the University of California, Eos Angeles, in 
the area of Education Psychology. 

Dr. Eilep is past president of the National Society for Programmed Instruction, and 
is a member of the AHRA, APA, AAAS, Phi Delta Kappa, and DAVE He is editor of, 
and contributor to, Piospectives in Programming ; a former department editor for AV 
Communication Review ; contributing editor to Educational Technology; and contributor 
to the le acher's Encyclopedia and the Annual Review of Information Science and Tech- 
nology . Dr. Eilep Is Biological Sciences Curriculum Study Special materials author and 
has also written programmed materials (text and CAI) in crystallography, spatial 
analysis of electrocardiograms, and biochemistry. He has authored articles in a number 
of professional publications. 
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ROBERT GERLETTi, Los Angeles County Superintendent of School^ Office. 

Director of the Division of Educational Media, is a graduate of the University of 
Southern California, and adjunct professor at that institution; Director of the Regional 
Educational Television Association of California; past president of several r 'te and 
national professional associations in the field of educational media. 

MRS. PATRICIA HARRISON, Educational Development Corporation. 

Mrs. Harrison is EDC’s Vice President for Production. Prior to joining EDC as 
Graphic Arts Director in 1966, Mrs. Harrison ran her own Design Studio in Palo Alto. 
She also directed graphics work for Varian Associates. Her designs for instructional 
materials have won several awards. 

EVAN R. KE1SLAU , Graduate School of Education, UCLA. 

Professor Keislar possesses a national reputation as an experimental researcher 
concerned with instruction and lear As a senior researcher, Pr. Keislar has chosen 
his areas of inquiry carefully during the past decade and, not surprisingly, much of his 
work has been related, directly or indirectly, to the area of educational product develop- 
ment. From the earliest days of SWRL's existence, for example, Dr. Keislar played a 
prominent role in development activities, at one time heading one of the Laboratory’s 
major projects. His continuing relationship with SWRL attests to his interest in educa- 
tional development as an enterprise amenable to analysis and improvement. 

Dr. Keislar is a Professor at the Gra ''Me School of Education, University of 
California, Los Angeles. He rt Jived h 1 B.A. in nhematics from the College of 
Pacific, his M,A. in Religious Education, om the 1 acific School of Religion, and his 
Ph.D. in Educational Psychology from the University of California. 

Dr. Keislar has been a Fellow for the Center for Advanced Study in the Behavioral 
Sciences, Stanford University, California, 1963-1964; an Assistant Professor, Tufts Col- 
lege, Medford, Oregon, 1946-1948; a Research Associate, College Entrance Examination 
Board, Princeton, New Jersey, 1945-1946; and an Instructor of Psychology, Princet< n 
University, 1946. He has also served as Consulting Elditor to the American Educational 
Res earch Journal and Review Editor, 1966-1967. He was a member of the Committee of 
Learning and the Educational Process, Social Science Research Council, 1962-1966. 

WALTER R, KINGSON , Theater Arts Department, UCLA. 

Dr. Kingson is a Professor on rhe Television -Film Faculty, University of California 
at Los Angeles at present, ard since 1949. He received his B.A. from the Uni v ersity of 
Wisconsin, 1939, M.A. from University of Wisconsin, 1940, did Graduate work at the 
University of Illinois, 1941 and at Columbia University, 1946; and his Ed.D. from New 
York University, 1948. It was the first doctor's degree granted unde Charles Siepmann 
in Communications in Education, 



At UCLA Dr. Kingson is in charge of the Workshop in Educational Broadcasting, 
offered for students, teachers and administrators who wish to use broadcast media in 
the classroom or for general purposes. 

He had done varied free-lance assignments for radio, television, and film in Holly- 
wood, and an active member of AFTRA and SAG since 1950. In 1969-70, Dr. Kingson was 
associate producer and director of ’’American Economic History" an innovative teaching 
project of the History Department, UCLA, using multimedia for college teaching. In 
1965-66, he was Fulbright Scholar Educational Television, at the University of London. 
In Spring of 1966 he was a visiting lecturer in educational television in Finland, Sweden, 
Norway, Denmark, France, and Germany. 
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Dr. Kingson was consultant in educational television at Television Espanola, the 
Spanish national broadcasting organization in 1966. In Fall of 1962, he was on a United 
Nationa (UNESCO) assignment as specialist in broadcasting to set up a Broadcast T rain- 
ing Centre in Israel. This involved planning and organizing curriculum, library, demon- 
stration - materials, staff, and physical plant to train new personnel for Kol Yisrael, the 
Israeli broadcasting organization. The Centre is now operating Successfully. 

Dr. Kingson was co-author of the following: Broadcasting Television and Radio, 

Prentice Hall, New York, 1955: Radio Drama Ac ti ng and Directing, Holt, Rinehart and 
Winston, New York, 1950. (Revised edition); Radir Drama Production, Rinehart & Co., 
New York, 1946; Television Directing and Performance, Holt, Rinehart and Winston, New 
York, 1965. 

HEX MALCOLM, BFA Educational Media. 

Twelve years experience in the Los Angeles Cit schools as a teacher of English and 
Social Studies, Curriculum consultant, television consultant, Audio-Visual Specialist; 
Associate Director of the Special Media Institute Project at USC; co-author of the Cali- 
fornia State adapted text The Earth; Maps and Globes, Noble & Noble; co-author of the 
Holt, Reinhardt, & Winston text. K nowing Our Neighbo r s in the Eastern Hemisphere ; 
in-service training consultant to various school districts; audiovisual consultant to 
various media producers; Director of Product Dev< ’opment, BFA Educational Media. 

JOHN D. McNE lL, Graduate School of Education, UCLA. 

Dr. John D. McNeil is Professor of Education, Graduate School of Education, Uni- 
versity of California, Los Angeles and a Co-director of the Instructional Objectives 
Exchange, a nonprofit educational corporation. His teaching has been concerned with two 
broad instructional objectives; (1) deepe r analysis of key concepts involved in the various 
branches of knowledge contributing to objectives and (2) development of research method- 
ology for experimental studies in the select?' and ordering of learning experiences. 
Many of these studies have been in the area of reading and include the identification and 
teaching of prerequisite skills for success in school. He has taught courses in founda- 
tions of education and elementary and secondary education as well as graduate courses 
and doctoral seminars in cuiriculum and supervision. McNeil has developed courses and 
experimental programs in teacher education for the military, Peace Corps, and adminis- 
trative leaders as well as numerous other projects involved in development activities. 
As one of three co-authors for a prominent reading series published by a major commer- 
cial firm, he is in a position to bring broad insights to the design of development and 
evaluation training programs. 

Dr. McNeil received his B, A. in English and M.A. in School Administration from 
San Diego State College. He received his Kd.F). in Curriculum and Teaching from Teachers 
College, Columbia University. 

Dr. McNeil's early association with programmed instruction research and with 
broader aspects of instructional materials preparation have led to a continuing in levels 
of educational development. He has been a participant in the UCLA Product Research 
Training Program since its organization four years ago. He has served ac a consultant 
for SWRL in public schools. His procedure for analysis, appraisaj and improvement of 
instruction, "Supervision by Objectives" is now used in many school systems. His 
publications include: Supervision; A Synthesis of Th ought and Actio n (Co-author), New 

York; McGraw Hill Book Company, Inc., 1969; Curr iculum Adm inistra t ion; Basic Prin- 
ci ples o f Curricul um I nstruction , New York; The Macmillan Company, 1965; co-author 
Read 'Se ries, American Book Company, 1968. 

MAYNAR D ORME , KCFT. 

Maynard Orme is director, Educational Services at KCM, He holds a B, * , in 
Music from U.C. Berkeley, and an M.A, in Iheater Arts at l’( 1 A, md is currcnrly 
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